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I.  INTRODUCTION 


A.  PURPOSE 

The  research  area  of  this  thesis  will  be  Modeling  and  Simulation  (M&S)  Policy  as 
it  pertains  to  the  economics  of  its  application  in  the  acquisition  discipline  of 
developmental  testing. 

The  US  Army  Developmental  Test  Command  (DTC)  has  the  missions  of 
developmental  technical  testing  and  safety  certification  of  all  Army  materiel.  A  test 
might  be  a  live  test  of  physical  hardware  in  a  physical  environment  or  a  virtual  test  of 
simulated,  and/or  physical  hardware  in  a  simulated  environment.  DTC  is  leveraging 
M&S  to  accomplish  its  missions  through  the  Virtual  Proving  Ground  (VPG)  Program. 
DTC  supplies  a  customer  decision-maker,  usually  a  Program  Manager  (PM),  with  data  on 
the  cost-effectiveness  of  new  virtual  and  physical  test  technologies.  DTC  requires  a 
methodology  to  develop  a  business  plan  that  supports  the  use  of  M&S  and  to  capture  the 
cost  avoidance  of  particular  virtual  tests  consistent  with  current  Army  cost  estimation 
procedures.  DTC  commissioned  independent  studies  of  past  test  programs  to  estimate 
the  costs  to  achieve  the  same  scope  of  testing,  as  tested  using  available  virtual  test 
techniques  and  as  using  previous,  less  VPG-intensive  test  methodologies.  The  studies 
showed  that  virtual  testing  provided  significant  cost  benefits  to  each  PM.  An  objective  is 
to  examine  cost  avoidance  results  from  those  studies  and  additional  data  with  a 
methodology  consistent  with  current  cost  estimation  guidance  to  determine  a  return  on 
investment  relationship. 
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B.  BACKGROUND 


The  earliest  use  of  modeling  can  be  traced  back  to  1867  with  the  Rigid  Kriegspiel 
(War  Game).  M&S  have  been  used  in  the  design  of  weapon  systems  for  many  years. 
The  testing  community  has  been  using  M&S  techniques  for  the  at  least  the  last  ten  years 
to  enable  testing  that  was  previously  not  possible  or  economical.  The  testing  community 
needs  a  consistent  technique  to  quantify  the  benefit  of  using  M&S.  This  benefit  can  then 
help  prove  that  sound  decisions  are  being  made  for  using  M&S.  More  particularly,  the 
developmental  test  community  needs  a  technique  since  they  are  integral  part  of  assessing 
whether  the  performance  of  a  weapon  system  is  mature  and  ready  for  production.  The 
developmental  testing  community  is  constantly  working  with  the  research  and 
development  community  that  is  using  M&S  extensively.  With  the  advent  of  the  use  of 
M&S  as  directed  by  Department  of  Defense  (DoD)  policy,  the  costs  and  associated 
benefits  with  the  development  and  use  of  M&S  techniques  fit  traditional  cost  benefit 
models  with  modification  and  much  emphasis  on  indirect  benefits.  Commonly,  the 
decision  to  employ  M&S  comes  as  a  result  of  policy  direction  rather  than  economic 
benefit.  There  is  no  documented  or  rigorous  technique  to  account  for  the  benefit  realized 
when  using  M&S  in  the  developmental  testing  process.  This  thesis  will  endeavor  to 
establish  an  equitable  methodology  for  accounting  or  realizing  the  direct  benefits 
associated  with  using  M&S  in  testing.  The  details  of  the  steps  will  be  developed  as 
necessary  to  perform  a  business  case  analysis. 

While  there  are  some  common  applications  industry-wide,  DoD  is  not  focused  on 
profit,  but  on  performance  and  total  ownership  costs.  The  1993  Government 
Performance  and  Results  Act  (GPRA)  has  put  pressure  on  PMs  by  putting  in  place  a 


results-based  management  regime  intended  to  tie  funding  decision  directly  to  program 
performance  (Laurent,  2000).  The  GPRA  requires  organizations  to  set  outcome  goals, 
measure  their  performance  and  report  their  accomplishments.  These  reports  to  be 
developed  by  PMs  must  be  coupled  with  cost  data  and  based  on  credible  information  to 
enable  decision-makers  to  make  funding  decisions.  The  GPRA  is  going  to  push  agencies 
to  clarify  their  business  lines  and  to  collect  performance  and  cost  information.  With  this 
information,  PMs  will  be  better  able  to  build  “better  business  cases  for  maintaining 
current  programs  or  moving  to  more  effective  methods  for  achieving  results”  (Laurent, 
2000).  A  recent  Industrial  College  of  the  Armed  Forces  (ICAF)  team  surveyed  a  large 
number  of  DoD  PMs  with  programs  in  various  stages  of  the  acquisition  process.  The 
survey  requested  general  information  from  the  PMs  regarding  how  they  made  decisions 
on  M&S  investments.  In  an  effort  to  maintain  balance,  they  also  surveyed  several 
Government  contractors.  Additionally,  they  also  conducted  a  literature  search  to  capture 
the  body  of  knowledge  in  non-Govenunent  organizations  related  to  justifying  M&S 
investments. 

The  ICAF  team  published  their  efforts  in  the  Fall  2000  issue  of  the  Acquisition 
Review  Quarterly.  The  article  provides  a  business .  case  framework  for  Program 
Managers  (PMs)  within  the  Department  of  Defense  to  use  when  determining  how  to 
apply  modeling  and  simulation  in  project  management  (Brown,  Grant,  Kotchman, 
Reyenga,  &  Szanto,  2000).  This  thesis  will  build  upon  this  framework  to  develop  one  for 
developmental  testing.  A  business  case  analysis  is  a  methodology  to  evaluate  investment 
opportunities.  M&S  employ  models,  either  statically  or  over  time,  to  develop  data  as  a 
basis  for  making  managerial  or  technical  decisions.  Models  are  physical,  mathematical 
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or  logical  representations  of  a  system,  entity,  phenomenon,  or  process  (DoD,  1997). 
Simulations  are  methods  for  implementing  models  over  time.  Normally,  we  associate 
simulations  with  a  software  program  that  implements  models  over  time,  within  the 
context  of  a  given  scenario.  Simulations  permit  the  user  to  assess  variables  and  the 
predictability  of  a  single  or  series  of  outcomes. 

The  use  of  M«&S  is  widely  misunderstood  within  the  DoD.  Recent  Government 
direction  to  use  Simulation  Based  Acquisition  (SBA)  in  DoD  programs  is  an  example  of 
a  policy  with  good  intentions  but  poorly  shaped  execution.  This  edict  has  been  met  with, 
at  best,  marginal  acceptance,  and  at  worst,  abject  resentment.  Such  resentment  and 
apprehension  spring  from  institutionalized  biases,  including  DoD  funding  procedures, 
that  work  against  optimizing  the  potential  gains  of  employing  modeling  and  simulation. 
By  far,  the  severest  criticisms  targeted  at  M&S  center  on  the  debate  over  Return  on 
Investment  (ROI).  ROI  is  just  one  of  many  techniques  for  evaluating  the  use  of  M&S  in 
a  program.  Ostensibly,  DoD  has  accepted  this  new  technology  as  a  means  of  reducing 
costs,  increasing  cost  avoidance  and  banking  the  residual  benefits  for  other  projects. 
Seldom  is  the  PM  or  testing  community  given  sufficient  funds,  staff,  or  time  to 
investigate  the  potential  benefits  of  tools  or  technologies  such  as  M&S.  Importantly, 
leadership  provides  little  incentive  to  capture  data,  build  expensive  models  or  conduct 
additional  analyses  to  transfer  M&S  results  to  other  projects.  Simply  put,  everyone  is 
under  intense  pressme  to  complete  his  or  her  programs  on  or  under  budget  and  within 
time  lines.  Existing  programs  lack  enticement  to  develop  new  models  or  simulation  tools 
that  may  have  wider  application  to  other  programs,  or  that  will  be  much  cheaper  to 
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operate  and  sustain.  With  few  exceptions,  these  occurrences  were  more  a  result  of 
coincidence  than  deliberateness. 

Perhaps  the  greatest  impediment  to  M&S  acceptance  is  lack  of  knowledge.  Many 
people  do  not  understand  the  potential  benefits  of  this  technology,  or  how  to  define  needs 
and  produce  the  right  tools  that  will  help  the  project.  Unfortunately,  this  reticence  may 
be  reinforced  by  an  institution  which  neither  favors  nor  rewards  risk  takers.  Sometimes, 
a  PM  may  not  know  for  certain  if  a  major  investment  in  M&S  is  warranted.  If  he  wants 
to  invest  in  the  technology  “just  to  see  if  there  is  a  benefit”  he  faces  criticism  if  the  results 
are  not  positive.  Thus,  he  must  weigh  the  costs  of  the  risk,  and  consequently  few  take  the 
chance,  preferring  more  traditional  approaches  like  the  building  of  expensive  mock  ups; 
the  use  of  labor,  time  and  money  intensive  trials,  and  incurred  costs  of  waste. 

While  all  of  this  bodes  well  for  an  anecdotal  argument  supporting  M&S,  what’s 
needed  is  a  more  reasoned  and  defensible  process  for  determining  M&S  investments. 
While  few  refute  the  intuitive  benefits  of  M&S,  PMs  quite  rightly  argue  that  any  tool 
must  be  first  measured  against  its  potential  benefits  before  it  is  used.  Unfortunately,  the 
question  is  not  easily  answered.  It  depends  on  a  variety  of  factors,  including  the  project’s 
funding,  time  period  to  recoup  the  investment,  and  perceived  benefits  to  developing  and 
using  M&S  in  the  program.  The  crux  of  the  problem  is  the  dilemma  of  costs  versus  risks 
and  the  potential  return.  Those  who  develop  and  use  M&S  need  a  methodology  to 
evaluate  investment  opportunities. 
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C.  RESEARCH  QUESTIONS 

1.  Primary  Research  Question 

The  main  question  of  the  thesis  may  be  stated  as:  “Can  a  rigorous  business 
analysis  be  applied  to  the  utilization  of  M&S  as  applied  to  developmental  testing? 

2.  Secondary  Research  Questions 

The  imderlying  questions  that  support  this  overarching  question  are: 

•  What  are  the  background,  history,  policy  and  guidance  related  to  using 
M&S  in  the  DoD  acquisition  process? 

•  What  are  the  detailed  steps  to  developing  an  M&S  business  case  analysis? 

•  How  must  this  business  case  be  tailored  for  application  to  developmental 
testing  and  what  are  the  results  when  applied  to  available  cases? 

•  What  conclusions  and  recommendations  may  be  drawn  from  the  preceding 
information  that  might  be  applied  to  M&S  efforts  in  support  of 
developmental  testing? 

D.  SCOPE  OF  THE  THESIS 

The  main  thrust  of  the  study  will  be  to  fully  develop  a  business  case  analysis 
process  for  assessing  the  benefit  of  using  M&S  in  the  developmental  testing  community. 
Additionally,  the  thesis  will  explore,  evaluate,  and/or  develop  methodologies  to  assess  the 
economic  benefit  of  using  M&S  to  enable  performance  of  developmental  testing  and 
determine  under  what  conditions  M&S  can  be  cost  effective.  Based  on  analysis,  it  will 
be  determined  if  cost  benefit  is  an  effective  measure  or  whether  other  extrinsic  factors 
determine  the  worth  of  using  M&S  in  performing  developmental  testing. 

E.  METHODOLOGY 

The  main  thrust  of  the  research  methodology  shall  be  to  apply  the  developed 
rigorous  business  case  to  several  developmental  testing/M&S  efforts.  This  research  will 
be  performed  through  analyzing  data  from  studies  performed  about  previous  VPG  efforts. 
This  business  case  will  be  developed  and  refined  through  application  to  a  system  that  was 
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tested  using  live  testing  techniques  and  the  application  of  VPG  M&S  techniques.  The 
motivation  to  take  this  approach  is  to  show  how  the  business  case  can  be  applied  prior  to 
program  initiation  to  ensure  that  the  economic  benefit  of  the  use  of  M&S  in  a  particular 
situation  can  be  assessed  accurately.  Usually  M&S  are  applied  where  possible  and  then 
its  worth  is  determined  after  completion.  This  technique  suggests  that  a  rigorous  business 
case  has  not  been  applied  in  the  past.  Once  the  analyses  have  been  performed,  then  the 
live  testing  cases  will  be  compared  to  determine  the  economic  benefit  of  using  M&S. 
This  methodology  is  intended  to  provide  credibility  to  the  business  case  for  using  M&S 
in  performing  developmental  testing. 

F.  ORGANIZATION 

This  thesis  will  be  organized  in  the  following  marmer: 

•  Introduction 

•  M&S  History,  Policy  and  Guidance 

•  Development  of  the  Business  Case  Framework 

•  STEP  1 .  Establish  a  Baseline 

•  STEP  2.  Establish  a  Vision  and  Direction 

•  STEP  3.  Quantify  the  Costs  and  benefits 
of  Altemative/Capabilities 

•  STEP  4.  Evaluate  Alternatives 

•  STEP  5.  Conduct  Sensitivity  Analysis 

•  STEP  6.  Develop  a  Migration  Strategy 

•  STEP  7.  Monitor  the  process  and  Continue  Feedback 

•  Applying  the  Business  Case  Framework  to  Developmental  Testing 

•  Conclusions  and  Recommendations 

G.  BENEFITS  OF  THE  STUDY 

This  thesis  is  timely  in  the  sense  that  the  DoD  has  created  an  initiative  over  the 

last  five  years  to  use  SBA  and  M&S  techniques  throughout  the  acquisition  process.  The 

limitation  of  the  SBA  effort  is  that  there  is  no  clear  direction  or  guidance  for 

implementation  of  its  techniques  in  a  cost-effective  manner.  However,  the  acquisition 
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community  has  been  constantly  required  to  prove  the  benefits  and  more  specifically,  the 
cost  benefits  derived  from  the  use  of  SBA.  The  thesis  sponsor,  US  Army  Developmental 
Test  Command,  will  benefit  from  the  methodology  developed  herein  to  better  predict, 
assess,  and  account  for  the  benefits  associated  with  using  M&S  in  their  testing  processes. 
Additionally,  the  Business  Case  Framework  will  provide  a  technique  that  will  better 
allow  documentation  of  M&S  decisions  that  can  withstand  inquiries.  This  thesis  will 
bring  critical  insights  into  the  process  necessary  to  determine  the  benefits  that  may  be 
derived  from  using  M&S  in  the  testing  process.  This  effort  will  add  to  the  body  of 
knowledge  for  the  application  of  M&S  in  the  acquisition  community. 
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II.  HISTORY  OF  M&S  INCLUDING  POLICY  AND  GUIDANCE 


A.  MODELS  AND  SIMULATIONS 

Since  the  beginning  of  time,  models  have  been  used  to  represent  abstract  and 
physical  phenomenon.  So  one  should  not  be  surprised  that  one  of  the  most  powerful 
tools  being  used  by  the  defense  acquisition  community  to  include  test  and  evaluation  is 
modeling  and  simulation.  Acquisition  professionals  are  finding  new,  unique  and 
challenging  applications  of  models  run  over  time  creating  a  simulation  of  a  system, 
subsystem,  or  the  environment  that  the  system  is  immersed  in.  It  is  indeed  possible  to 
create  a  synthetic  environment  where  systems,  both  real  and  virtual,  can  be  tested  to 
determine  the  system’s  proper  performance  for  its  intended  mission. 

The  word  model  brings  to  mind  several  mental  images.  Some  remember  the 
plastic  pieces  of  an  airplane,  car  or  ship;  all  connected  in  an  orderly  fashion  to  be  pulled 
apart,  assembled,  trimmed  and  sometimes  painted.  Others  may  remember  watching  an 
old  war  movie  where  the  maneuver  plans  were  drawn  in  the  dirt  or  scale  mockups  used 
for  mission  rehearsals.  A  model  is  defined  as  a  physical,  mathematical  or  otherwise 
logical  representation  of  a  system,  entity,  phenomenon  or  process.  A  simulation  can  have 
two  definitions:  a  method  for  implementing  a  model  over  time;  and  a  technique  for 
testing,  analyzing,  or  training  in  which  real-world  systems  are  used,  or  where  real-world 
and  conceptual  systems  are  reproduced  by  a  model. 

The  Army  has  used  explicit  representation  of  combat  systems,  combat  and  other 
processes  for  a  number  of  years.  Long  before  the  advent  of  computers,  the  Army  relied 
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heavily  upon  information  derived  from  the  conduct  of  simulations.  History  contains 
examples  of  planning  and  rehearsing  missions  using  sand  tables,  and  developing  force 
structure  and  tactics  using  substitutes  for  weapons  and  weapon  systems  during  the 
Louisiana  Maneuvers  of  1940.  The  sophistication  of  tools  used  to  model  and  simulate 
combat  systems  and  combat  processes  evolved  over  the  years.  Many  major  field  and 
command  post  exercises  were  conducted  using  probability  tables  and  the  rolling  of  die  to 
simulate  the  occurrence  of  events.  Rapid  advances  in  computer  technology  sped  the 
evolution  of  M&S  into  the  synthetic  environment. 

During  the  evolution  from  predominantly  physical  representations,  the  Army’s 
use  of  M&S  continued  to  support  a  variety  of  applications  for  five  major  pmposes: 
education,  training,  and  military  operations;  analysis;  research  and  development;  test  and 
evaluation;  and  production  and  logistics. 

Since  World  War  II,  technology  has  advanced  at  an  ever-increasing  rate.  This 
explosion  of  technology  is  moving  faster  than  the  acquisition  community  can  acquire 
products.  With  technology  we  can  accomplish  what  was  once  considered  impossible. 
With  the  utilization  of  microprocessors,  many  new  spin-off  technologies  are  enabling 
new  designs  everywhere.  Virtual  reality  with  an  interactive,  computer-generated  or 
synthetic  environment  is  significantly  changing  our  lives;  entertainment,  work,  learning, 
travel  and  commimications  are  all  incorporating  virtual  reality.  Today,  information  is 
being  moved  instead  of  people. 

B.  M&S  HISTORY,  POLICY  AND  GUIDANCE 

The  following  will  give  a  brief  history  of  modeling  and  simulation  and  the  policy 
and  guidance  that  direct  its  use  in  the  acquisition  environment.  Table  1  in  Appendix  A 
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contains  a  detailed  chronological  history  of  M&S  and  the  contributing  world  technology 
events  that  influenced  its  development  and  application. 

Until  as  recently  as  the  mid-1980s,  the  Army’s  development  and  use  of  M&S 
were  accomplished  on  an  as-needed  and  as-afforded  basis.  In  the  late  1980s,  the  advent 
of  distributed  simulation  technology,  led  by  the  Advanced  Research  Projects  Agency 
(ARP A)  introduced  the  Army  to  Simulation  Network  (SIMNET).  The  SIMNET,  coupled 
with  a  dovmward  trend  in  defense  budgets  led  the  Army  to  seek  M&S  application 
simultaneously  addressing  more  than  one  of  the  purposes  mentioned  previously.  In 
distributed  simulation  technology,  the  Army  recognized  the  potential  for  linking  M&S  of 
various  types,  fidelities,  and  resolutions,  and  of  establishing  these  linkages  from 
geographically  separated  sites  both  in  the  Continental  United  States  and  overseas.  In 
addition,  the  Army  was  assigned  the  role  of  executive  agent  for  DoD  in  developing  the 
technology  and  infrastructure  to  support  military  applications  of  Distributed  Interactive 
Simulation  (DIS). 

The  Army  introduced  management  of  its  models  and  simulations  in  the  early 
1980s  with  Training  and  Doctrine  Command  (TRADOC)  Regulation  5-4.  Today,  the 
Army  executes  management  of  its  M&S  through  the  Army  Modeling  and  Simulation 
Office  (AMSO)  with  policy  prescribed  in  Army  Regulation  AR  5-11.  Army  M&S  used 
in  the  T&E  process  receives  its  vision  through  the  Developmental  Test  Command’s 
Virtual  Proving  Ground  Master  Plan. 

In  Jxme  1991,  the  Deputy  Secretary  of  Defense  approved  a  plan  to  strengthen  the 
use  of  modeling  and  simulation  (M&S).  He  also  designated  the  Under  Secretary  of 
Defense  for  Acquisition  and  Technology  (USD(A&T)),  as  responsible  for  strengthening 
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the  use  of  M&S  in  joint  education  and  training,  research  and  development,  test  and 
evaluation  and  operation  and  cost  analysis.  In  June  1992,  the  Institute  for  Defense 
Analyses  (IDA)  published  a  report  titled  “A  Review  of  Study  Panel  Recommendation  for 
Defense  Modeling  and  Simulation.”  IDA  reviewed  179  recommendations  made  by  25 
separate  study  panels,  over  a  16-year  period,  concerning  defense  M&S.  The  Defense 
Modeling  and  Simulation  Office  (DMSO),  using  this  document  as  a  conceptual 
foundation,  reviewed  and  classified  the  recommendations  and  set  off  to  plan  for  and 
implement  those  that  provided  for  new  and  extended  applications  for  M&S.  The  DMSO 
especially  focused  on  systems  acquisition  and  test  and  evaluation.  The  initiative 
instituted  by  the  Deputy  Secretary  of  Defense  is  now  referred  to  as  the  Defense  Modeling 
and  Simulation  Initiative. 

Some  key  conclusions  drawn  firom  the  IDA  study  were:  1)  Specific  areas,  such  as 
the  architectural  issues  of  interoperability  and  specification  of  standards,  and  the  life- 
cycle  support  of  defense  models  and  simulations,  deserve  more  attention  and  support;  2) 
There  are  many  areas  to  which  defense  M&S  either  should  be  applied  anew  or  extended, 
especially  those  associated  with  systems  acquisition;  3)  There  are  substantial  needs  and 
opportunities  for  improving  management  and  coordination  of  defense  M&S  activities. 
Particularly  of  interest  to  this  work  is  the  fact  that  there  is  a  lack  of  life-cycle  support  and 
methodologies  to  determine  the  life-cycle  costs  of  defense  models  and  simulations. 

Then,  in  Jime  1993,  Dr.  Anita  Jones,  the  Director  Defense  Research  and 
Engineering,  established  the  Acquisition  Task  Force  on  Modeling  and  Simulation 
(ATFMS)  to  recommend  actions  which  would  lead  to  the  more  effective  and  integrated 
use  of  modeling  and  simulation  throughout  the  acquisition  process.  The  final  report  of 
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the  ATFMS,  dated  June  1994,  concludes  that  1)  the  effective,  integrated  use  of  M&S  in 
the  acquisition  process  is  being  impeded  by  the  lack  of  an  overall  M&S  architecture;  2) 
the  lack  of  this  M&S  architecture  has  led  to  not  getting  the  most  out  of  the  investment 
that  is  being  made  in  M&S  for  acquisition;  3)  clearly  designated  leadership  and  an 
appropriate  coordinating  mechanism  are  required  to  stimulate  progress  in  the  creative 
application  of  M&S  to  the  acquisition  process;  4)  additional  education  and  training  on  the 
capabilities  and  limitations  of  M&S  are  required  for  all  participants  in  the  acquisition 
process;  and  5)  it  is  time  to  apply  advanced  M&S  and  related  tools  to  enhance  real 
acquisition  programs.  (Parker,  1 994)  The  recommendation  of  the  ATFMS  to  develop  an 
overall  M&S  architecture  spurred  the  initiation  of  the  development  of  the  High  Level 
Architecture  (HLA)  within  the  DoD. 

Beginning  in  1994,  the  Army  implemented  a  policy  that  requires  all  Army 
acquisition  strategies  for  ACAT  I  and  II  programs,  subsequently  expanded  to  Advanced 
Technology  Demonstrations  (ATD),  to  include  a  Simulation  Support  Plan  (SSP).  In  this 
plan,  the  PM  must  lay  out  the  functional  requirements  for  M&S  to  support  engineering 
and  combat  developments,  test  and  evaluation,  training,  and  military  exercises  to  support 
the  program.  The  PM  must  also  develop  an  M&S  acquisition  strategy  identifying 
resources  required  bringing  the  M&S  to  fruition. 

Through  the  Army  M&S  Management  process  M&S  standards  categories  have 
been  developed  to  enable  the  use  of  M&S  throughout  the  three  Army  M&S  domains; 
Training,  Exercises  and  Military  Operations  (TEMO),  Advanced  Concepts  and 
Requirements  (ACR),  and  Research,  Development  and  Acquisition  (RD&A).  M&S  that 
are  developed  at  scientific  and  engineering  levels  are  included  in  the  RD&A  domain. 
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Developmental  Test  and  Evaluation  using  M&S  falls  within  the  RD&A  domain. 
According  to  AR  5-11,  Management  of  Army  Models  and  Simulations,  M&S  developed 
for  use  in  Test  and  Evaluation  will  be  managed  over  their  life  cycles  in  accordance  with 
the  responsibilities  defined  in  AR  70-1  and  AR  73-1.  (AR  5-11,  1997)  Within  the  Army 
Standards  Categories,  there  is  a  category  called  Cost  Representation.  This  category 
includes  the  data,  tools,  algorithms  and  techniques  necessary  for  accurately  costing  and 
consistently  portraying  all  aspects  of  activities  portrayed  in  models  and  simulations.  The 
Cost  Representation  standards  category  currently  lists  two  requirements:  develop 
methods  to  cost  all  elements  portrayed  in  M&S  and  standardize  techniques  for  comparing 
costs  of  alternatives.  To  date  however,  this  standards  category  group  has  not  published 
anything  to  address  their  two  listed  categories.  This  group  should  be  followed  for  future 
cost  representation  information.  This  group  is  lead  by  the  Director  of  the  United  States 
Audit  Agency  (USAA)  Cost  and  Economic  Analysis  Center,  Ruth  Johnson. 

On  Sep  10,  1996,  Paul  Kaminski  signed  out  the  DoD  High  Level  Architecture 
(HLA)  for  Simulations  Policy  which  designates  the  HLA  as  the  standard  technical 
architecture  for  all  DoD  simulations.  (Kaminski,  1996)  The  HLA  has  been  designated  as 
the  standard  technical  architecture  for  all  DoD  simulations.  This  mandate  for  HLA 
compliance  supersedes  all  previous  requirements  for  DoD  simulations  to  comply  with 
other  simulation  standards  such  as  DIS  or  Aggregate-Level  Simulation  Protocol.  All 
Army  simulations  will  meet  DoD  standards.  HLA  establishes  a  common  high-level 
simulation  architecture  to  facilitate  the  interoperability  of  all  types  of  simulations  among 
themselves  and  with  Command,  Control,  Communications  and  Intelligence  systems.  The 
HLA  will  also  facilitate  the  reuse  of  M&S  as  the  Army  moves  into  an  era  of  federations 
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of  simulations  producing  synthetic  battle  environments  across  all  domains  and  inclusion 
of  M&S  into  all  major  acquisition  programs. 
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III.  DEVELOPMENT  OF  THE  BUSINESS  CASE  FRAMEWORK 


A.  RESULTS  OF  SURVEYS 

A  survey  conducted  in  1999  by  an  ICAF  team  indicated  that  PMs  are  investing  in 
M&S  to  support  program  development;  however,  “most  of  the  investment  decisions  were 
based  on  intuition  or  need-based  factors.  Most  decisions  were  made  without  detailed 
quantitative  analysis  (Brown  et  al.,  2000)”.  For  the  most  part,  PMs  accepted  modest 
investment  in  M&S  because  they  believed  that  such  investments  would  benefit  the 
program.  Most  of  these  efforts,  however  commendable,  usually  lacked  the  use  of  a 
methodical  cost/benefit  analysis.  “The  lack  of  a  structured  business  case  analysis  made  it 
difficult,  if  not  impossible,  for  PMs  to  articulate  or  substantiate  their  investment  strategy” 
(Brown  et  al.,  2000).  In  some  cases  PMs  developed  and  explored  M&S  capabilities 
without  even  knowing  what  benefits  might  be  achieved.  Often  the  PM  is  willing  to  take  a 
risk  in  using  M&S  in  the  development  of  his  program  with  the  hopes  that  a  marginal  yet 
unquantified  benefit  will  be  realized.  The  PMs  in  the  ICAF  study  were  “embracing  new 
technology;  however,  without  a  sovmd  business  case  approach,  they  could  not  really 
assess  if  the  investment  would  realize  considerable  benefits  or  generate  prohibitive  costs” 
(Brown  et  al.,  2000).  Most  PMs  justified  their  M&S  investment  based  on  one  or  more  of 
the  following:  reducing  design  cycle  time;  augmenting  or  replacing  physical  tests; 
helping  resolve  limitations  of  funds,  assets  or  schedules;  or  providing  insight  into  issues 
that  were  impossible  or  impracticable  to  examine  in  other  ways.  (Brown  et  al.,  2000) 

Much  of  the  feedback  from  the  ICAF  study  reflected  that  PMs  had  tried  to 
examine  and  measure  the  costs  and  benefits  of  using  M&S  but  without  the  aid  of  any 
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business  case  analysis  format.  Of  the  few  reported  to  have  used  a  disciplined  approach, 
approaches  and  results  varied  among  PMs  surveyed.  This  use  of  inconsistent 
applications  or  approaches  leads  to  mixed  results  that  cannot  be  readily  compared  or 
evaluated.  ‘The  majority  of  respondents  suggested  the  question  of  using  M&S  was  not 
one  of  “Should  I”  but  rather,  “How  can  I?”  This  demonstrates  general  acceptance  that 
M&S  is  required  in  an  efficiently  managed  program,  but  it  suggests  the  question,  “At 
what  cost?’”  (Brown  et  al.,  2000)  In  the  cases  where  the  cost  investment  is  prohibitively 
expensive  or  the  long-term  benefits  are  marginal,  the  decision-maker  should  select  from 
alternatives,  be  it  “foregoing  M&S,  partnership  with  other  PMs  to  share  costs,  or 
leveraging  the  investment  of  others.  This  is  an  important  decision,  given  the  DoD’s 
current  fiscal  climate.”  (Brown  et  al.,  2000) 

PMs  want  “confirmation  that  investment  in  M&S  will  yield  direct  savings  within 
their  budgets.”  (Brown  et  al.,  2000)  Considering  that  much  of  the  benefit  of  M&S 
investment  is  intangible,  traditional  cost  benefit  meastirement  approaches  may  not 
provide  an  accurate  assessment.  In  this  situation,  PMs  may  dismiss  using  M&S  tools  if 
the  estimated  ROI  is  not  significant  for  their  immediate  project  needs.  However,  since 
some  projects  are  relatively  short-lived,  the  longer-term  residual  benefits  may  very  well 
be  missed  due  to  short  sightedness  on  the  part  of  PMs.  While  traditionally  the  benefits  of 
M&S  tend  to  be  discussed  in  terms  of  ROI,  several  alternatives  for  business  case  analysis 
can  just  as  effectively  justify  M&S  investments.  “The  challenge  is  to  define  an 
appropriate  strategy  and  priorities  to  address  the  business  value  proposition.”  (Brown  et 
al.,  2000) 
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A  disciplined  business  case  approach  and  methodology  has  many  benefits.  “It 
can  help  bring  the  aggregate  benefits  into  focus  and  strengthen  the  argument  for  M&S 
investment.”  (Brown  et  al.,  2000)  A  business  case  analysis  provides  a  convenient 
mechanism  for  project  management  to  explore  investment  possibilities  and  assess  the 
potential  risks.  It  can  be  an  easy-to-follow,  logical  thread  that  lays  the  groundwork  for 
current  and  future  programs  to  attain  useful  information,  metrics  and  measures.  “This 
approach  forces  a  timeline,  captures  benefits  and  enables  authorities  to  decide  if  the 
return  is  worth  the  investment  in  simulation  technologies.”  (Brown  et  al.,  2000) 

B.  MAKING  THE  BUSINESS  CASE 

The  business  case  entails  addressing  seven  basic  areas  in  the  prescribed  order. 

•  Establish  a  Baseline 

•  Establish  a  Vision  and  Direction 

•  Quantify  the  Costs  and  Benefits  of  Altemative/Capabilities 

•  Evaluate  Alternatives 

•  Conduct  Sensitivity  Analysis 

•  Develop  a  Migration  Strategy 

•  Monitor  the  Process  and  Continue  Feedback 

(Brown  et  al.,  2000) 

Corporate  America  is  taking  a  methodical  approach  to  investment  decisions 
regarding  everything  to  include  M&S.  Although  these  methodological  approaches  are 
“primarily  profit  driven,  they  share  common  maxims  of  production,  such  as  cost 
reduction,  efficiency  and  cost  avoidance,  to  that  of  DoD.”  (Brown  et  al.,  2000) 
Essentially,  a  business  case  will  assist  the  -decision-maker  in  evaluating  which  of  a 
number  of  logical  alternatives  will  best  meet  the  objective. 
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1.  Step  1.  Establishing  a  Baseline 

The  first  step  in  this  decision-making  process  is  to  establish  an  accurate  baseline. 
The  key  in  determining  the  success  or  failure  of  a  project,  program  or  function  is 
establishing  its  current  state  before  any  changes  are  considered  or  implemented.  “The 
baseline  provides  a  benchmark  from  which  decisions  will  be  weighed  and  assessed.” 
(Brown  et  al.,  2000)  Performance  measures,  regardless  of  type,  must  have  a  baseline 
measurement  to  show  the  changes/improvements  a  project  undergoes  as  it  achieves  it 
goal.  Establishing  a  baseline  measure  is  essential  in  establishing  the  validity  of  a 
performance  measure.  A  baseline  measure  is  usually  the  first  measure  taken  of  a  system 
or  project.  Subsequent  measurement  may  be  defined  as  the  new  baseline  if  substantial 
changes  to  the  system  make  the  earlier  baseline  obsolete  or  reduce  the  effectiveness  of 
the  particular  performance  measure.  “The  baseline  should  include  a  clear  enunciation  of 
assumptions  and  constraints.”  (Brown  et  al.,  2000) 

Assumptions  are  explicit  statements  describing  the  present  and  future 
environment.  They  reduce  complex  situations  into  manageable  proportions.  These 
assumptions  normally  provide  some  indication  of  the  estimated  future  workload,  the 
useful  life  of  the  investment  or  system,  and  the  period  of  time  over  which  alternatives 
will  be  compared.  Assumptions  should  also  discuss  sunk  costs  and  realized  benefits,  but 
are  not  included  as  part  of  the  baseline.  Constraints  are  those  factors  that  limit 
alternatives.  Normally  they  are  expressed  in  terms  of  time,  finances,  institutional  or 
regulatory  statutes  or  directives  and  physical  plant/assets. 

The  baseline  must  identify  the  high  value  portions  of  a  program  in  order  to 
evaluate  the  appropriateness  of  various  alternatives.  In  other  words,  the  decision-maker 
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must  look  at  the  areas  of  the  program  which  will  take  the  largest  portion  of  program  time 
in  which  to  invest  M&S  resources  to  achieve  a  time  benefit.  Additionally,  the  decision¬ 
maker  could  also  address  which  areas  of  the  program  are  most  costly  and  then 
appropriately  apply  M&S  resources  to  potentially  achieve  a  cost  savings/benefit. 
Following  an  informed  examination  of  the  program  being  analyzed,  the  PM  would  be 
better  able  to  allocate  M&S  spending  where  it  would  offer  the  greatest  potential  savings 
or  benefits.  The  baseline  must  determine  these  high  value  areas  for  an  effective  business 
case  analysis  to  be  performed.  These  program  specifics  form  the  drivers  to  the  program, 
which  in  turn  drive  the  investment  process.  (Brown  et  al.,  2000) 

2.  Step  2.  Establishing  Vision  and  Direction 

The  decision-maker  “must  look  to  the  futme,  then  bridge  the  gap  between  present 
knowledge  and  that  required  to  make  future  products  a  reality.”  (Brown  et  al.,  2000)  The 
acquisition  decision-maker  or  PM  must  establish  the  program's  vision  for  using  M&S. 
As  the  PM  is  developing  his  plan  for  use  of  M&S  in  support  of  his  acquisition  program, 
he  should  keep  focused  to  ensure  success.  The  PM  should  use  the  list  of  probing 
questions  contained  in  the  SB  A  “Cheat  Sheet”  which  is  provided  in  Appendix  B.  A 
performance  plan  needs  to  be  established  for  the  program  that  should  contain 
performance  goals  and  measures.  Performance  goals  should  be  expressed  in  an  objective, 
quantifiable,  and  measurable  form.  Performance  measures  should  be  established  that  are 
indicators  for  measuring  or  assessing  the  relevant  service  levels,  outcomes  or  outputs  and 
comparing  actual  program  results  with  the  established  performance  goals. 

The  performance  plan  for  the  program  must  identify  the  performance  goals  and 
measures  that  are  based  on  the  goals  and  objectives  of  the  agency’s  strategic  plan. 
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Performance  goals  that  are  representative  of  milestones  in  the  achieving  the  long-term 
objectives  contained  in  the  organizational  strategic  plan  should  be  included  in  the 
performance  plan.  The  performance  plan  should  establish  the  performance  measures  or 
indicators  to  be  pursued  in  measuring  or  assessing  the  relevant  service  levels,  outcomes 
or  outputs  and  comparing  actual  program  results  with  the  established  performance  goals. 
Additionally,  the  performance  plan  will  describe  the  operational  processes,  skills  and 
technology,  and  the  human,  capital,  information,  or  other  resources  required  to  meet  the 
performance  goals.  The  performance  plan  will  provide  a  basis  for  comparing  actual 
program  results  with  the  established  performance  goals;  and  describe  the  means  to  be 
pursued  to  verify  and  validate  the  measured  values. 

For  some  programs,  there  may  be  a  gap  between  the  capabilities  provided  by 
existing  resoiuces,  including  M&S  resources  and  program  goals/objectives  as  stated  in 
program  performance  plans.  Some  performance  gaps  may  be  resolved  by  reengineering 
processes  with  or  without  the  use  of  M&S.  If  reengineering  of  a  process  is  needed  it 
should  be  done  prior  to  determining  what  M&S  investments  are  needed  to  support  the 
redesigned  process. 

A  functional  requirement  analysis  should  be  performed  to  determine  the 
requirements  that  the  M&S  investment  must  meet  to  fill  the  performance  gap.  The 
analysis  should  identify:  the  performance  criteria,  goal,  or  ultimate  ouqjut,  a  definition  of 
the  common  uses  of  the  M&S  investment,  a  ranking  of  the  requirements  in  order  of 
importance,  and  a  decomposition  of  functional  requirements  into  self-contained  features 
(GSA,  1998).  To  allow  flexibility  in  evaluating  various  solutions,  functional 
requirements  should  not  be  described  in  equipment  and  software  terms,  but  in  terms  of: 
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business  outcome,  mission,  purpose,  capability,  organizational  program  components 
involved,  schedule  and  cost  objectives,  and  operating  constraints.  The  minimum 
information  that  should  be  identified  by  an  analysis  of  functional  requirements  for  an 
M&S  system  includes:  system  functional  description,  inputs/outputs,  processes,  data 
characteristics,  performance  criteria  (accuracy,  validation,  timing,  flexibility),  interfaces, 
failure  contingencies  and  security  specifications. 

Wherever  possible,  requirements  for  M&S  systems  should  be  stated  using  an  open 
system  architecture.  The  open  system  architecture  should  encompass  the  following 
characteristics:  user  applications  not  tied  to  a  single  hardware  or  system  software 
manufacturer,  new  functionality  added  from  different  contracts  without  significant  effort, 
other  systems  tied  into  or  interfaced  with  the  system  without  significant  effort,  and  the 
system  fits  the  organizational  M&S  architecture  plan.  Other  agencies  that  may  have 
acquired  systems  to  accomplish  similar  goals  should  be  identified  and  management 
should  look  for  cross-agency,  other  M&S  domain  solutions  or  Government  wide 
economies  to  avoid  duplication  of  effort. 

As  with  other  developing  technologies,  “the  vision  should  consider  how  M&S 
tools  can  improve  program  costs,  scheduling  and  performance,  and  whether  scientific 
knowledge  exists  to  support  such  M&S  investment.”  (Brown  et  al.,  2000)  The  vision 
should  drive  or  focus  what  the  M&S  tools  should  be  trying  to  solve,  not  the  other  way 
around.  The  vision  statement  clearly  identifies  what  M&S  capabilities  are  to  be 
achieved,  without  dictating  how  they  will  be  accomplished. 

During  this  program  vision/direction  planning  effort,  determining  the  functional 

requirements,  feasibility,  alternative,  cost,  and  benefits  must  be  accomplished  as 
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efficiently  as  possible.  An  Integrated  Product  Team  (IPT)  should  be  formed  to  help 
ensure  the  program’s  success  throughout  its  life  cycle.  The  IPT  will  assist  the  program 
manager  in  completing  and  documenting  the  analyses  needed  for  each  project  life  cycle 
phase.  The  IPT  for  projects  involving  procurements  must  include  procurement 
specialists,  financial  analysts,  M&S  experts  and  weapon  system  experts.  The  IPT  will 
help  identify  functional  requirements  that  describe  functional  and  customer  needs  that 
must  be  satisfied  by  the  M&S  investment.  The  IPT  must  perform  market  research  to 
ascertain  if  the  market  can  provide  the  desired  models  and  assets  necessary  to  meet  the 
program  requirements.  This  market  research/analysis  is  designed  to  produce  a  list  of 
alternatives,  with  accompanying  data  necessary  to  assess  affordability,  benefits  and  costs. 

Market  research  is  done  before  the  development  of  any  formal  requirements 
documents  are  developed.  The  following  information  should  be  collected  during  the 
initial  market  survey:  1)  Availability  of  commercial  items  or  other  Government  assets 
available  to  meet  the  need,  and  whether  they  might  require  modification;  2)  The 
customary  practices  and  the  associated  costs  for  customizing/modifying  items  to  meet  the 
need;  3)  The  customary  practices  regarding  warranties  and  discounts  for  the  identified 
products;  4)  The  laws  and  regulations  which  may  apply  to  the  acquisition  of  identified 
products;  and  5)  The  distribution  and  support  capabilities  of  possible  suppliers.  The 
decision  on  the  contract  type  should  be  taken  fi-om  the  results  of  the  market  research. 
Once  the  vision  and  direction  have  been  set  and  the  requirements  have  been  drafted,  the 
cost  and  benefits  of  each  alternative  discovered  during  the  market  research  must  be 
quantified. 
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3.  Step  3.  Quantify  the  Costs  and  Benefits  of  Alternatives/Capabilities 

0.  Introduction 

Since  alternatives  are  logical  packages  of  initiatives  that  work  well 
together  (Kidwell,  1998),  one  alternative  is  the  status  quo.  This  alternative  is  often 
identified  during  the  development  of  the  baseline  in  Step  1.  In  some  cases,  the  baseline, 
often  a  physical  prototype  or  live  test,  can  be  more  cost  effective  than  the  use  of  M&S 
depending  on  the  configuration  or  technology  employed.  “Other  alternatives  should 
represent  various  combinations  of  M&S  tools  that  help  achieve  the  vision.”  (Brown  et  al., 
2000) 

In  the  conceptual  stages  of  a  project,  some  of  the  detailed  information 
about  benefits  and  costs  associated  with  different  alternatives  may  not  be  available.  For 
this  reason,  the  benefit  and  cost  analysis  information  should  be  updated  and  corrected  as 
necessary  as  the  project  planning  proceeds  to  its  later  phases  and  more  information 
becomes  available. 

In  determining  alternatives,  the  decision-maker  must  consider  both  the 
immediate  and  long-term  effects  of  each  proposal.  “First,  what  does  the  program  need  in 
order  to  perform  better?  Second,  what  does  the  program  need  to  enable  its  survival  imtil 
the  next  stage?”  (Brown  et  al.,  2000)  Ultimately,  a  program  must  satisfy  all  requirements 
while  at  the  same  time,  it  must  endure  and  survive  each  step  of  the  process  without  being 
cut.  “It  does  no  good  for  a  program  to  bankrupt  itself  with  massive  unfocused  M&S 
investment  early  on.”  (Brown  et  al.,  2000)  Each  investment  must  produce  value  during  a 
timeframe  that  is  appropriate  for  the  program  to  achieve  its  performance  goals.  In  order 
to  survive,  the  PM  must  meet  his  near-term  milestones  in  order  to  survive;  therefore,  he 
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must  create  a  program  that  can  withstand  the  tests  of  time.  The  alternatives  must  also 
consider  the  technological  advances  in  M&S  tools  that  might  occur.  “A  program  might 
not  be  justified  in  spending  huge  sums  of  money  on  M&S  technology  that  may  be 
superceded  and  rendered  obsolete  in  2-5  years.”  (Brown  et  al.,  2000) 

The  decision-maker  when  performing  a  cost  benefit  analysis  must  identify 
all  costs  that  are  incident  to  achieving  each  alternative.  M&S  can  be  expensive  to 
develop,  particularly  in  domains  where  the  scientific  principles  are  not  fully  understood 
as  applied  to  the  problem.  While  additional  research  can  fill  the  knowledge  voids,  the 
cost  of  this  research  must  be  factored  into  the  analysis  of  alternatives.  These  should 
include  the  opportunity  costs  of  assets  and  resources,  which  are  the  alternative  value 
foregone  when  an  asset  is  used  for  other  purposes.  (DoDI  7041.3,  1995)  They  also 
include  non-recurring  and  recurring  costs.  Life  cycle  costs  should  include  all  costs,  non¬ 
recurring  and  recurring  that  occur  over  the  life  of  an  alternative.  (GSA,  1998) 

b.  Benefit  Cost  Analysis  (BCA)  Framework 

Consistent  with  Office  of  Management  and  Budget  (0MB)  Circular  A-94 
guidance,  a  BCA  should  encompass  and  address  the  following  elements:  explicit 
imderlying  assumptions  used  to  arrive  at  the  estimates  of  future  benefits  and  costs, 
evaluation  of  alternative  means  for  achieving  program  objectives,  and  plans  for  periodic, 
results  oriented  evaluation  of  the  actual  costs,  benefits,  and  program  effectiveness 
attributable  to  the  investment. 

c.  BCA  steps 

The  BCA  process  encompasses  the  following  steps:  (1)  Identify 
assumptions  and  constraints,  (2)  Identify  alternatives  and  their  schedules,  costs  and 
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benefits,  (3)  Evaluate  alternatives,  (4)  Perform  risk  and  sensitivity  analysis,  (5)  Develop 
performance  goals  and  measures  for  monitoring  the  project. 

The  first  two  steps  of  the  BCA  process  apply  to  this  step  of  the  business 
case  analysis.  The  rest  of  the  BCA  steps  will  be  applied  to  the  latter  steps  of  the  business 
case  analysis. 

(1)  Assumptions  and  Constraints.  Assumptions  are  explicit 
statements  used  to  describe  the  present  and  future  environment  upon  which  the 
benefit/cost  analysis  is  based.  The  purpose  of  assumptions  is  to  reduce  complex 
situations  to  problems  of  manageable  proportions. 

0MB  Circular  A-94  requires  aneilyses  to  be  explicit  about 
underlying  assumptions  to  arrive  at  estimates  of  future  benefits  and  costs  and  include  a 
statement  of  the  assumptions,  the  rationale  behind  them,  and  a  review  of  their  strengths 
and  weaknesses.  Examples  of  assumptions  include  estimated  future  workload,  estimated 
useful  life  of  an  investment  or  system,  and  the  period  of  time  over  which  alternatives  will 
be  compared.  Constraints  are  factors  external  to  the  relevant  environment  that  limits 
alternatives  to  problem  resolution.  They  may  be  physical,  time  related,  financial,  or 
institutional/regulatory.  They  provide  boundary  limits  for  the  alternative  solutions  to  a 
particular  problem. 

(2)  Identifying  and  Estimating  the  Benefits  and  Costs  of 
Alternatives.  Some  examples  of  simulation  related  alternatives  are:  do  nothing,  use 
Commercial  Off  the  Shelf  (COTS)  packages  or  purchase  new  software  or  equipment, 
modify  existing  hardware/software,  develop  new  software,  and  purchase  services. 
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One  alternative  that  should  always  be  considered  is  continuing  the 
status  quo.  Each  alternative  will  have  its  own  mix  of  resources.  Costs  must  be  identified 
and  itemized  at  a  level  of  detail  consistent  with  the  budgeting  process.  Alternatives  will 
also  have  different  benefits  realization  periods  and  some  additional  or  direct  benefits.  As 
the  project  or  procurement  process  proceeds,  the  BCA  and  budget  requests  will  be 
updated  to  reflect  the  most  current  information  on  alternatives  based  on  the  project  and 
procurement  progress. 

The  estimate  of  costs  and  benefits  of  an  investment  or  project 
should  show  the  difference  that  results  fi’om  making  the  investment;  specifically  the 
change  in  cash  flows  as  a  result  of  imdertaking  the  project.  Basic  questions  to  be  asked 
are: 

•  What  additional  funds  will  be  required  to  carry  out  the  chosen  alternative? 

•  What  additional  revenues  will  be  created  over  and  above  any  existing 
ones? 

•  What  costs  will  be  added  or  removed  as  a  result  of  the  investment? 

Benefit-cost  analysis  for  M&S  investments  compares  the  costs  of 
the  M&S  investment  or  project  (whether  it  be  a  new  system,  a  replacement  system, 
system  enhancement,  or  a  hardware/software  purchase)  to  the  savings  derived  from  the 
expected  business  and  operational  improvements  resulting  from  the  M&S  investment  or 
project.  The  basic  elements  of  cost  comparison  are  the  total  M&S  investment^system  and 
business  costs  if  the  system  is  implemented  versus  the  total  system  and  business  costs  if 
the  system  were  not  implemented  or  if  the  current  system  is  continued.  The  savings 
resulting  from  the  system  implementation  are  associated  with  tangible  benefits. 
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Additional  intangible  benefits  are  also  documented  and  considered  in  the  decision  to 
approve  system  development. 

(3)  Tangible  and  Intangible  Benefits  and  Costs.  Consistent 
with  0MB  Circular  A-94,  both  tangible  and  intangible  benefits  and  costs  should  be 
recognized.  The  relevant  cost  concept  is  broader  than  private-sector  production  and 
compliance  costs  or  Government  cash  expenditures.  Costs  should  reflect  the  opportunity 
costs  of  any  resources  used,  measured  by  the  return  to  those  resources  in  their  most 
productive  application  elsewhere.  A-94  provides  additional  guidance  on  identifying  and 
measuring  benefits  and  costs  including:  incremental  benefits  and  costs  and  sunk  costs, 
transfers,  indirect  measures  of  benefits  and  costs,  multiplier  effects,  treatment  of 
inflation,  discount  rates,  and  lease  purchase  analysis. 

After  the  decision  has  been  made  that  it  is  beneficial  for  an  agency 
to  acquire  the  use  of  a  capital  asset,  0MB  Circular  A-94  guidelines  should  be  used  to 
perform  a  Lease-Purchase  analysis  to  determine  if  the  agency  should  purchase  or  lease 
the  asset.  Lease-purchase  analyses  should  compare  the  net  discounted  present  value  of 
the  life  cycle  cost  of  leasing  with  the  full  costs  of  buying  or  constructing  an  identical 
asset. 

(4)  Identifying  and  Estimating  Costs.  When  considering  the 
costs  of  projects/altematives,  one  must  take  into  consideration  the  project’s  total  life 
cycle  cost,  as  defined  in  0MB  Circulars  A-94  and  A- 109  to  include  all  acquisition  costs 
and  the  cost  of  operations. 

(5)  Project  Life.  It  is  often  difficult  to  estimate  the  life  of  a 
project.  The  accepted  criterion  is  the  continued  ability  to  generate  satisfactory  cash  flows 
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or  other  intangible  benefits.  The  period  of  time  over  ’which  the  savings  or  benefits  to  be 
gained  fi-om  a  project  may  be  expected  to  accrue  is  the  economic  life  of  a  project.  The 
economic  life  is  generally  the  lesser  of  physical  life,  technological  life  or 
mission/product-market  life.  The  mission  life  is  that  period  of  time  over  which  a  need  for 
the  asset  or  program  is  anticipated.  The  physical  life  is  the  period  during  which  a  facility 
or  piece  of  equipment  ’will  be  available  for  use.  The  technological  life  is  the  period  of 
time  before  which  improved  technology  would  make  an  asset  obsolete.  The  project  life 
of  the  M&S  investments  sometime  occur  several  years  prior  to  the  time  the  project  starts 
providing  benefits.  This  elapsed  time  period  between  initial  fimding  and  the 
commencement  of  the  economic  life  is  referred  to  as  lead-time.  Project  life  consists  of 
the  total  of  the  lead-time  and  the  economic  life. 

(6)  Methods  of  Alternative  Comparison.  Economic  lives  and 
lead  times  can  vary  among  alternatives.  The  following  guidelines  are  recommended  for 
determining  the  comparison  period.  If  both  the  economic  lives  and  lead  times  for  all 
alternatives  are  the  same,  there  is  no  problem  as  the  comparison  ’will  be  between  the  same 
project  life.  In  the  case  where  the  same  economic  lives  and  different  lead  times,  the  first 
year  that  expenditures  must  be  made  for  any  one  of  the  alternatives,  should  be  considered 
the  base  year  or  first  year  for  all  the  alternatives.  When  dealing  with  different  economic 
lives,  one  method  is  to  let  the  economic  life  of  the  dominant  asset  prevail  ’with  subsidiary 
assets  replaced  as  necessary.  Another  method  when  dealing  ’with  different  economics 
lives  is  to  use  the  shortest  economic  life  and  impute  residual  value  in  the  asset  with  the 
longer  life.  Because  of  the  inherent  uncertainties  of  making  estimates  in  distant  years,  in 
some  cases  it  may  be  necessary  to  set  arbitrary  limits  on  the  planning  horizon  to  be  used 


in  the  analysis.  This  planning  horizon  can  be  shorter  than  the  estimated  economic  life  of 
the  project. 


(7)  Cost  categories.  Cost  categories  to  consider  when 
estimating  the  cost  of  an  altemative/project  include  the  categories  following,  which  are 
not  mutually  exclusive. 

•  Research  and  development:  These  costs  are  often  associated  with  the 
development  of  new  simulation  systems  and  include  items  such  as  model 
development  or  test  bed  costs. 

•  Investment:  Investment  costs  are  essentially  one-time  costs  and  include 
costs  such  as:  land,  new  construction,  rehabilitation,  equipment,  software 
purchases,  system  development  (functional  requirements,  design,  analysis, 
programming,  testing,  conversion),  and  relocation. 

•  One-time  personnel  cost  (recruitment,  separation,  training,  travel,  etc.) 

•  Value  of  Existing  Assets  Employed:  This  is  the  value  of  existing  assets. 
This  value  is  included  in  the  investment  cost  only  when  the  existing  asset 
is  currently  in  use  on  some  other  project,  or  was  intended  for  sale. 

•  Terminal/residual  value:  This  is  the  expected  value  of  buildings, 
equipment  or  other  assets  at  the  end  of  their  economic  lives  and  is  treated 
as  a  reduction  in  the  life  cycle  cost  of  the  particular  alternative  for  which 
the  use  of  the  asset  is  intended.  Residual  value  is  the  computed  value  of 
assets  as  any  point  in  time.  Residual  value  may  or  may  not  coincide  with 
terminal  value.  Terminal/residual  value  should  be  applied  to  existing  asset 
replaced  as  well  as  new  assets  being  acquired. 

•  Operation  and  Maintenance  Costs:  These  costs  occur  continually  over  the 
useful  life  of  the  project.  They  include  labor  costs  of  operating  and 
maintenance  personnel,  fuel  and  power  costs,  operating  and  maintenance 
supply  costs,  spare  and  repair  parts  costs,  insurance  costs,  taxes,  and  a 
share  of  indirect  (overhead/burden)  costs.  These  costs  can  be  substantial 
and  occur  over  time  until  the  structure,  system  or  equipment  is  retired 
from  service. 

•  Variable  costs:  These  are  a  group  of  costs  that  vary  in  some  relationship  to 
the  level  of  operational  activity  (such  as  direct  labor,  direct  material) 

•  Fixed  costs:  These  are  a  group  of  costs  that  do  not  vary  with  output. 

•  Total  cost:  This  is  the  sum  of  all  life  cycle  costs  associated  with  the 
product/system. 

•  Unit  cost:  This  is  the  total  cost  divided  by  some  related  base  and  may  be 
expressed  in  terms  of  cost  per  item  produced,  per  person,  etc.  Unit  cost 
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represents  an  average  that  may  change  with  the  magnitude  of  the 
numerator,  denominator,  or  both. 

•  Recurring  and  nonrecurring  costs.  Recurring  costs  are  those  costs  that 
occur  from  one  period  to  the  next  at  specified  intervals;  whereas,  non¬ 
recurring  costs  are  one-time,  non-repetitive  costs.  Life  cycle  costs 
embrace  all  costs,  non-recurring  and  recurring  that  occur  over  the  life  of 
an  alternative. 

•  Sunk  Cost:  These  are  costs  that  have  already  been  incurred  and  cannot  be 
recovered  or  altered  by  future  action.  They  are  irrelevant  to  the  benefit- 
cost  analysis  because  only  the  future  consequences  of  investment 
alternatives  can  be  affected  by  current  decisions. 

For  the  more  avid  reader,  the  definitions  of  the  cost  factors  that 
apply  to  these  cost  categories  can  be  found  in  Appendix  C. 

(8)  Cost  Estimating  Techniques.  A  thoroughly  reasoned 
benefit-cost  analysis  requires  the  collection  of  financial  information  called  cost  elements 
from  budget  documents  along  with  estimates  of  proposed  simulation  investment/system 
costs.  The  selection  of  cost  estimating  techniques  depends  on  the  amount  and  detail  of 
available  data  and  the  time  and  resources  available  to  develop  the  cost.  The  required 
level  of  effort  for  the  different  estimating  techniques  ranges  from  extreme  analytical 
detail  to  intuition.  The  three  cost  estimating  techniques  are  the  bottom-up  method, 
parametric  analysis  method  and  the  analogy  method. 

The  industrial  engineering/bottom-up  method  consolidates 
estimates  from  several  separate  work  segments  into  a  total  project  estimate.  It  involves 
segmenting  the  total  product  into  single  parts  for  which  detailed  estimates  can  be 
established.  Where  detailed  data  exists,  the  industrial  engineering  approach  can  result  in 
extremely  detailed  and  complete  estimates. 

The  parametric  cost  estimating  or  parametric  analysis  method 
focuses  on  what  the  project  is  supposed  to  accomplish  or  yield  compared  to  similar 
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projects.  The  yield  or  benefits  form  the  basis  or  parameters  for  the  cost  estimates.  Once 
these  benefits  and  their  measures  are  established,  relationships  between  the  parameters 
and  their  costs  are  developed,  mainly  form  historical  data.  This  method  is  used  when 
data  is  inadequate  for  employing  the  industrial  engineering  approach.  It  is  also  a 
preferred  method  for  deriving  cost  estimates  at  the  earliest  stages  of  development. 

The  analogy  method  uses  judgment,  specifically  analogies,  which 
are  direct  comparisons  with  similar,  historical  systems  or  products.  This  method  requires 
expertise  and  intuitive  reasoning.  When  little  historical  data  is  available  and  neither  the 
industrial  engineering  nor  parametric  methods  can  be  used,  the  analogy  method  is  used 
but  it  is  seldom  the  most  accurate. 

In  developing  cost  data  for  a  life  cycle  cost  analysis,  one  should 
initially  investigate  possible  data  sources  to  determine  what  is  available  for  direct 
application  to  analysis  objectives.  If  the  required  data  are  not  available,  the  use  of 
parametric,  cost  estimating  techniques  may  be  appropriate.  Existing  data  banks,  initial 
system  planning  data,  supplier  documentation,  reliability  and  maintainability  predictions, 
test  data  should  all  be  investigated  a  potential  data  sources. 

(9)  Identifying  and  estimating  benefits.  All  benefits  resulting 
from  each  alternative  should  be  identified.  Both  quantifiable  and  non-quantifiable 
benefits  should  be  identified  and  described.  Determining  benefits  is  the  most  difficult 
part  of  the  benefit/cost  analysis  because  it  is  often  difficult  to  identify  all  benefits  and 
accurately  quantify  and  monetize  them.  0MB  Circular  A-94  suggests  the  principle  of 
willingness-to-pay  to  obtain  a  given  benefit  and  that  market  prices  are  a  good  place  to 
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start.  To  the  extent  possible,  benefits  should  be  expressed  in  quantifiable  terms  and 
clearly  linked  to  the  program  goals  and  needs  identified  in  previous  planning  stages. 

Most  benefits  will  be  in  terms  of  improvements  in  effectiveness, 
efficiency,  or  customer  satisfaction.  Example  areas  of  benefits  include:  operating 
efficiency,  reliability/maintainability,  accuracy  manageability,  availability  service  life, 
quality  ecology,  economy  morale,  safety  security,  and  regulatory  compliance.  Examples 
of  types  of  quantifiable  and  monetizable  benefits  are:  reduced  resource  requirements 
(such  as  support  services,  supplies,  personnel,  training,  lease,  rental,  maintenance, 
computers),  improved  data  entry  (resulting  in  reduced  staff  time,  lowered  error  rates), 
improved  operational  effectiveness  (resulting  in  reduced  error  rates,  improved  timeliness, 
increased  productivity,  better  quality  products),  cost  avoidance  (by  eliminating  futme 
staff  growth,  minimizing  penalties  for  delays,  elimination  additional  equipment 
requirements). 

d.  Benefit-Cost  Analysis  Framework. 

The  decision  to  undertake  a  simulation  investment  or  project  is  based  on 
the  assumption  that  the  business  improvements  resulting  from  the  system  exceed  the 
costs  of  modifying  business  operations  and  maintaining  the  current  simulation  or  system 
(if  it  exists).  Benefit-cost  analysis  makes  explicit  the  assumed  business  rationale  that 
justifies  investments  in  simulation  versus  live  systems.  Benefit-cost  analysis  has  four 
major  elements:  total  business  and  system  costs  with  the  simulation  investment/new 
system,  total  business  costs  without  the  simulation  investment/new  system,  tangible 
benefits,  and  intangible  benefits. 


The  determination  of  costs  and  tangible  benefits  is  based  upon  five  basic 


cost  elements: 


•  Business  costs  with  the  simulation  investment  new  system— The  total  costs 
to  carry  out  the  business  functions  and  processes  to  be  automated  by  the 
simulation  investment/system. 

•  Business  costs  without  the  simulation  investment/new  system— The  total 
costs  that  would  be  incurred  to  continue  the  business  functions  and 
processes  vdth  the  current  level  of  automation  (which  may  be  no 
automation). 

•  Nonrecurring  costs  of  the  new  system-One  time  expenditures  that  will  be 
incurred  in  the  design,  development  or  acquisition,  and  implementation  of 
the  new  system.  These  expenses  will  not  be  incurred  after  a  system  is 
operational. 

•  Recurring  costs  of  the  simulation  investment/new  system— Ongoing 
expenses  that  will  continue  throughout  the  investment’s/system's  life 
cycle.  Most  of  these  costs  -will  be  incurred  during  the  operational  phase  of 
the  system. 

•  Costs  to  continue  the  current  system  (if  there  is  one)  —The  expenditures 
that  would  be  made  if  the  organization  continued  to  operate  the  existing 
system  (these  may  include  recurring  and  non-recurring  costs). 

Business  costs  are  presented  as  a  total  budget  projection  for  the  business 

operations  affected  by  the  proposed  simulation  investment/system.  Analysis  of  business 

costs  should  consider  the  same  factors  that  are  applied  in  developing  multi-year  budget 

projections.  The  estimates  for  the  benefit-cost  analysis  should  be  comparable  to  those 

produced  in  other  budget  exercises. 


Costs  of  a  simulation  investment/system  are  the  costs  required  to  design, 
acquire,  develop,  implement,  and  operate  the  simulation  investment/system.  These  are 
the  costs  related  to  the  simulation/system  itself  and  not  the  business  functions  supported 
by  the  simulation/system.  Costs  include  both  business  and  system  costs  with  and  without 
the  simulation  investment/system.  This  cost  comparison  quantifies  the  financial  impacts 
of  a  "go"  or  "no  go"  decision. 
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The  cost  of  operating  the  business  without  the  simulation/system 
highlights  the  investment  managers  would  be  forced  to  make  in  maintaining  current 
business  practices  and  system  operations.  The  cost  of  operating  the  business  with  the 
new  or  enhanced  system  highlights  the  tangible  bottom  line  payoff  of  the  proposed 
system.  Total  costs  with  the  simulation  investment  or  new  information  system  will,  in 
most  cases,  be  more  than  continuing  cxurent  operations.  Savings  can  accrue  in  the 
business  operations  that  exceed  the  additional  costs  associated  with  design,  development, 
acquisition,  and  maintenance  of  the  simulation  investment/information  system  itself  over 
a  projected  life  cycle. 

Tangible  benefits  can  be  measured  as  specific  cost  savings  to  the 
organization.  Tangible  benefits  are  the  cost  savings  resulting  fi'om  changes  in  business 
and  system  operations.  Each  item  in  the  cost  analysis  that  has  a  projected  saving  must  be 
associated  with  an  operational  change  that  will  produce  the  reduction  in  projected 
expenditures.  For  example,  the  cost  of  continuing  operations  without  the  simulation 
investment/new  system  may  include  the  hiring  of  additional  technical  staff  to  continue  to 
manually  process  projected  increases  in  workload.  If  the  proposed  simulation 
investment/system  were  implemented,  technology  would  replace  these  manual  processes 
and  no  additional  personnel  would  be  hired. 

Intangible  benefits  are  difficult  to  measure  in  financial  terms.  Despite 
their  lack  of  financial  rationale,  they  may  be  sufficient  to  justify  the  system  independent 
of  cost.  In  the  Federal  environment,  compliance  with  legal  and  regulatory  requirements 
is  an  intangible  benefit  that  can,  on  its  own,  justify  the  investment  in  information  and 
simulation  systems.  Other  examples  of  intangible  benefits  are  improved  customer 
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satisfaction,  faster  service,  and  increased  employee  job  satisfaction.  Intangible  benefits 
must  be  supported  by  a  clear  link  to  specific  outcomes  of  system  implementation. 

(1)  Benefit-Cost  Model  Components.  This  section  presents  a 
discussion  of  the  information  collection  requirements  for  the  proposed  benefit-cost 
model.  It  is  organized  around  identifying  fully  allocated  current  simulation/system  costs 
and  proposed  project  expenses.  Growth  values  such  as  present  value  calculations,  cost  of 
money  and  interest  expense  are  factored  into  this  model.  The  model  is  expressed  as 
simply  as  possible  and  is  consistent  with  Federal  guidelines  governing  benefit-cost 
analysis. 

Information  collection  is  intended  to  reflect  a  mixture  of  cost 
projections  and  assumptions  that  system  owners  have  gained  through  operating 
experience.  Information  concerning  new  proposed  systems  should  be  organized  and 
recorded  on  the  project  worksheet  (Figure  1.  Benefit-Cost  Calculations).  Backup 
documentation  to  this  cost  worksheet  should  be  explained  in  a  narrative  form  sufficient  to 
clarify  assumptions  about  the  numbers.  Complete  information  describing  the  current 
systems  should  also  be  provided  along  with  any  foreseen  benefits  from  continuing  current 
operations. 
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CALCULATION 

FORMULA 

RESULT 

Business  Savings 

Business  Costs  Business  Costs  Present 

Without  the  -  With  The  New  =  Business  x  Value  = 

New  M&S  System  M&S  System  Savings  Factor 

(a)  Present 

Value  of 

Business 

Savings 

Net  System  Costs 

Recurring  +  Costs  of  Net  Present 

Non-Recurring  -  Continuing  =  System  x  Value  = 

Costs  of  New  Old  System  Costs  Factor 

M&S  System 

(b)  Present 

Value  of  Net 
System  Costs 

Benefits 

Present  Present  Value 

Value  of  -  of  Net  M&S  System  = 

Business  Costs  (b) 

Savings  (a) 

1  (c)  Net  Present 
Value  of 

Benefits 

Benefit-Cost  Ratio 

Present  Present 

Value  Value  of  = 

of  Net  System 

Benefits  (c)  Costs  (b) 

Resent  Value 
Benefit-Cost 

Ratio 

Figure  1 .  Benefit-Cost  Calculations.  (GSA,  2000) 

The  model  is  designed  to  allow  comparison  of  costs  with  and 
without  the  simulation  investment/new  system;  non-recurring  costs,  recurring  costs, 
residual  values  and  benefits  over  an  estimated  systems  life.  Each  benefit-cost  worksheet 
should  be  carefully  analyzed  to  ensure  completeness  in  capturing  fully  allocated  cost 
projections. 


The  cost  worksheet  should  be  completed  for  the  full  life  cycle  of 
the  proposed  simulation  project/system.  The  life  cycle  includes  design,  acquisition, 
development,  implementation,  maintenance,  and  disposal.  Supporting  documentation 
should  identify  the  expected  time  period  for  each  life  cycle  stage  of  the  system.  A  six 
year  system  life  cycle  is  assumed  in  this  methodology.  Six  years  reflects  the  impact  of 
rapidly  changing  simulation  technology  on  the  useful  life  of  systems.  Some 
simulation/systems  may  have  a  longer  or  shorter  life  cycle.  In  these  instances  the  benefit- 
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cost  analysis  should  include  the  rationale  for  the  life  cycle.  The  methodology  and 
supporting  worksheets  can  be  adjusted  to  fit  the  projected  system  life  cycle. 

(2)  Business  Operation  Costs.  This  category  of  cost  elements 
identifies  the  total  costs  to  carry  out  the  business  functions  first  with  the  current  level  of 
automation  and  then  when  they  are  automated  by  the  simulation  investment/new  system. 
The  costs  need  to  be  calculated  both  with  and  without  the  proposed  new  simulation 
system.  Business  operations  are  the  activities  and  resources  used  to  conduct  the  functions 
to  be  supported  by  the  proposed  simulation  system.  Business  operation  costs  are  defined 
by  the  following  cost  factors:  personnel  salaries  and  fiinge  benefit,  supplies/equipment, 
facility  space  occupancy,  utilities,  maintenance,  travel,  training,  incidentals,  interagency, 
and  other  identified  costs. 

(3)  Nonrecurring  Costs  of  the  Simulation  Investment/New 
System.  This  category  of  cost  elements  identifies  the  one-time  expenditures  that  are 
incurred  in  the  design,  acquisition,  development,  and  implementation  of  the  new 
simulation  system.  Nonrecurring  costs  of  the  new  system  are  defined  by  the  following 
cost  factors:  conversion  costs,  replacement  or  upgrade  systems,  hardware,  software, 
communications,  contracting,  travel,  training,  studies,  parallel  operations,  incidental 
expenses,  residual  value,  and  other  identified  costs. 

(4)  Recurring  Costs  of  the  Simulation  Investment/System. 
This  category  of  cost  elements  identifies  the  ongoing  expenses  that  are  incurred  to 
maintain  and  operate  the  simulation/system  after  implementation  is  completed.  Costs 
should  be  projected  for  the  entire  useful  life  of  the  system:  parallel  operations,  hardware 
lease  or  rental,  software  lease  or  rental,  communications,  other  equipment,  facility  space 
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occupancy,  utilities,  maintenance  supplies,  personnel  salaries  and  fringe  benefits, 
security,  travel,  training,  system  testing  and  back-up,  incidentals,  interagency,  and  other 
identified  costs. 

(5)  Costs  of  Continuing  the  Existing  Simulation  System.  This 
category  of  cost  elements  identifies  the  simulation  system  costs  that  would  be  incurred  if 
the  existing  level  of  automation  or  simulation  were  continued  instead  of  developing  the 
new  system.  The  costs  of  continuing  the  old  simulation/system  are  defined  by  the 
following  cost  factors:  hardware  lease  or  rental,  software  lease  or  rental,  communications, 
other  equipment,  facility  space  occupancy,  utilities,  maintenance,  supplies,  personnel 
salaries  and  fringe  benefits,  security,  travel,  training,  system  testing  and  back-up, 
incidentals,  interagency,  and  other  identified  costs. 

(6)  Calculated  Values.  In  order  to  compare  costs  and  benefits 
at  a  point  in  time,  present  value  tables  are  used  in  the  model.  0MB  Circular  A-94  defines 
the  standard  criterion  for  deciding  whether  a  Government  program  can  be  justified  on 
economic  principles  as  the  net  present  value  of  benefits.  Net  present  value  is  the 
projected  savings  resulting  from  the  program  reduced  by  the  net  investment  required 
developing  and  implementing  the  program.  Using  the  costs  of  business  operations  and 
systems  described  above,  five  basic  calculations  are  made  to  assess  the  benefits  that  will 
result  from  the  new  system.  These  calculations  are  described  below  and  in  Figure  1 . 

Business  Savings.  Business  savings  is  the  difference  between  the 
cost  of  business  operations  without  the  simulation  investment/new  system  and  the  cost  of 
business  operations  with  the  new  system.  The  result  of  this  calculation  is  the  business 
savings  of  the  new  system. 


40 


Net  System  Costs.  Net  system  costs  is  the  difference  between  the 
cost  to  continue  to  operate  the  old  simulation/system  (if  there  is  some  existing 
simulation/system)  and  the  cost  to  design,  acquire,  develop,  implement  and  maintain  the 
simulation  investment/new  system.  Net  simulation/system  costs  are  the  additional 
investment  that  the  organization  will  make  in  the  simulation  investment/new  system  or,  if 
the  simulation  investment/new  system  is  less  costly,  the  system-related  savings  that  will 
result  from  the  new  system. 

Present  Value  Factor.  Cost  information  is  adjusted  in  the  model 
using  tables  to  multiply  costs  and  benefits  by  discount  factors  consistent  vvith  0MB 
Circular  A-94  guidance  to  determine  the  present  value  based  on  the  year  of  occurrence. 
Note:  0MB  Circular  A-94  is  updated  periodically.  Those  conducting  a  BCA  should 
ensure  that  they  are  using  the  proper  discount  rates,  consistent  with  the  latest  version  of 
0MB  Circular  A-94,  by  checking  the  Internet  at 
http://www.whitehouse.govAVH/EOP/OMB/html/circulars/. 

Net  Present  Value  (Of  Benefits).  Net  Present  Value  is  the  business 
savings  increased  or  decreased  by  the  net  simulation/system  costs.  If  the  new  system 
reduces  system  costs,  tangible  benefits  will  be  higher  than  business  savings.  If  the  new 
system  represents  increased  costs,  tangible  benefits  Avill  be  less  than  the  business  savings 
of  the  new  system. 

Benefit-Cost  Ratio.  The  Benefit-Cost  Ratio  is  the  value  of  the 
tangible  benefits  compared  to  the  net  simulation/system  costs.  Benefits  will  generally 
exceed  costs  and  this  ratio  will  almost  always  be  greater  than  1.  If  a  system  has  a  benefit- 
cost  ratio  less  than  1,  it  must  be  entirely  justified  by  its  intangible  benefits. 
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(7)  Benefits.  New  simulation  systems  provide  opportunities 
for  a  broad  range  of  improvements  to  technical  business  operations.  Not  all  benefits  of 
automation  will  result  in  a  tangible  benefit  that  reduces  costs.  Intangible  benefits  can  be 
an  important  factor  in  deciding  to  proceed  with  the  development  of  a  simulation  system. 
Intangible  benefits  should  be  documented  as  part  of  the  benefit-cost  analysis  and  included 
in  the  narrative  that  describes  the  proposed  system.  Intangible  benefits  should  be 
considered  with  the  benefit-cost  ratio  for  determining  the  rationale  for  continuing  with 
the  proposed  system.  In  identifying  tangible  and  intangible  benefits  of  the  new  system, 
the  following  should  be  considered: 

Reliability  Improvements.  The  benefit  gained  in  the  reduced  risk 
of  system  malfunction  or  failure,  and  reduced  downtime  for  manual  operations  versus  a 
comparison  system,  for  performing  the  same  or  equivalent  tasks. 

Accuracy  Improvements.  The  benefit  gained  in  process 
simplification  and  streamlining.  Ease  of  data  input  and  accuracy  rates  that  reduce  overall 
errors  are  reported  here. 

Labor  Productivity  Improvements.  The  benefit  gained  in 
performing  the  same  functions  and  tasks  for  fewer  hours  of  personnel  time.  These 
improvements  may  allow  staff  to  work  on  other  activities,  but  do  not  result  in  an  actual 
reduction  in  personnel. 

Grade  of  Service  Productivity  Improvements.  The  benefit  gained 
in  performing  a  service  more  efficiently  or  effectively  to  the  direct  benefit  of  the 
warfighter  or  taxpayers. 
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Compliance  With  Legal  And  Regulatory  Requirements.  The 
benefit  gained  by  meeting  procedural  or  performance  guidelines  specified  in  laws  and 
regulations.  Physical  environmental  limitations  can  often  be  overcome  by  stimulating  the 
live  system  with  a  simulated  environment  created  with  a  model. 

Customer  Satisfaction.  The  benefit  can  be  in  terms  of  a  reduction 
in  time  spent  analyzing  data  and  an  ability  to  answer  questions  with  more  accuracy  in  a 
timelier  manner. 

e.  Quantify  the  Cost  and  Benefits  of  Alternatives/Capabilities 
Reprise 

Identifying  alternatives  can  be  the  most  difficult  portion  of  the  process. 
Benefits  must  be  viewed  primarily  in  terms  of  measurable  value.  Expected  benefits 
should  flow  from  the  clear  operating  vision  developed  in  Step  2  of  the  business  case. 
Enigmatically,  there  are  both  quantifiable  and  non-quantifiable  benefits.  The  former 
usually  have  some  tangible  or  readily  identified  returns;  the  latter  have  less  so. 
Additionally,  there  may  be  benefits  that  have  no  intrinsic  value  to  one  program  but 
provide  value  to  others  which  are  called  external  benefits. 

(1)  Quantifiable  Benefits.  These  include  cost  savings,  time 
improvement,  acceleration  of  deliverables,  quality  enhancement  and,  in  most  cases,  cost 
avoidance  that  is  directly  related  to  the  program.  The  alternatives  must  also  consider 
existing  systems  and  programs.  If  improvements  are  to  be  measured  from  an  “as-is 
baseline”  -  then  it  is  not  advisable  to  start  from  ground  zero.  It  may  be  possible  to  look 
at  related  M«feS  domain  initiatives  in  other  programs,  assess  their  applicability,  and 
leverage  them  for  success.  The  cost  associated  with  using  these  existing  alternatives 
should  be  less,  given  that  a  majority  of  the  investment  would  be  a  sunk  cost  borne  by  the 


43 


previous  developers.  Similarly,  it  also  advisable  to  consider  partnering  with  another 
program  and  thereby  sharing  costs.  This  option  is  a  possible  alternative  to  reducing  the 
up-front  investment  required  for  a  new  development  effort.  One  should  ensure  that  they 
have  examined  all  potential  benefits  by  using  published  references  and  experts  in  the 
field  of  cost  analysis. 


(2)  Non-Quantifiable  Benefits.  Traditionally,  the  issues  of  risk 
reduction,  organizational  efficiency,  technology  transference,  product  safety,  and 
environmental  impact  reductions  have  been  considered  as  non-measurable  and  therefore 
non-quantifiable.  However,  these  are  important  issues,  and  one  must  consider  them  in 
their  analysis.  To  illustrate  this  point,  technology  transference  will  be  addressed. 

M&S  technology  transference  can  significantly  influence  costs,  but 
in  today’s  DoD  environment  it  has  yet  to  receive  adequate  attention.  Given  the  shrinking 
public  purse  and  the  demand  for  greater  accountability  and  responsibility  for  the  dispersal 
of  frmds,  all  must  show  due  diligence  in  their  public  spending.  They  must  consider  the 
residual  benefits  of  technology  transference. 

Some  M&S  investment  might  be  of  use  to  other  projects  and  Program 
Managers.  For  example,  the  Grizzly  PM  invested  heavily  in  chassis  M&S 
to  support  short-term  design  and  performance  analysis.  This  M&S 
investment  resulted  in  $21  million  worth  of  quantifiable  benefits  to  the 
Grizzly  program.  The  Grizzly  PM  funded  the  M&S  effort  through  internal 
reallocation  of  frmds.  The  PM’s  supervisor,  PM  Combat  Mobility 
Systems,  recognized  the  potential  for  the  use  of  these  models  for  both 
other  program  requirements  and  in  other  programs  sharing  the  Grizzly’s 
chassis.  He  leveraged  the  Grizzly  Program’s  M&S  investment,  securing 
funding  to  expand  the  applicability  of  the  initial  investment  into  other 
programs  and  to  support  other  long-term  Grizzly  requirements.  (Brown  et 
al.,  2000) 


When  forecasting  near-term  savings  in  design  and  production 


costs,  one  company  in  the  ICAF  study  accrued  substantial  non-quantifiable  benefits. 
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“The  engineers  made  a  substantial  leap  in  M&S  knowledge  when  learning  how  to  define 
data  needs,  how  to  shape  models,  and  how  to  refine  simulation  runs,  to  narrow  the 
bandwidth  of  problem  solving.  The  resulting  expertise,  data  and  process  could  be  applied 
to  future  projects.”  (Brown  et  al.,  2000)  Not  surprisingly,  the  company’s  models  and 
databases  are  the  envy  of  the  industry  based  on  their  visionary  efforts.  These  efforts 
helped  the  company  achieve  a  competitive  advantage  that  was  a  non-quantifiable  gain. 

Unfortunately,  many  decision-makers  and  PMs  dismiss  the  concept 
and  viability  of  non-quantifiable  benefits.  “Organizations  rarely  track  non-quantifiable 
benefits  or  those  outside  the  program's  realm  with  any  real  vigor.  They  don't  afford  them 
reasonable  weight  when  analyzing  costs  and  their  alternatives.”  (Brown  et  al.,  2000) 
Similarly,  external  benefits  almost  always  exist.  Usually,  these  benefits  that  are 
applicable  to  the  Government  and  DoD  at  large  are  not  acknowledged  due  to  the  not 
invented  here  mentality. 

External  Benefits  are  benefits  which  do  not  bring  direct  return  or 
savings  to  the  unique  program  being  managed,  but  have  applicability  to  other 
organizations.  As  mentioned  previously,  many  M&S  initiatives  and  their  products  can 
either  be  modified  or  directly  transferred  to  other  programs.  Looking  again  to  the  Grizzly 
program,  the  contractor  supporting  the  PM,  United  Defense  Limited  Partnership, 
developed  a  common  product  model  database  that  benefited  efforts  at  Aberdeen  Proving 
Ground,  The  Army  Warfighting  Analysis  and  Integration  Center,  Waterways  Experiment 
Station  and  National  Training  Center  projects  (Brown  et  al.,  2000).  This  example  shows 
that  there  is  a  residual  savings  or  external  benefits  for  follow-on  users.  PMs  have 
identified  problems  with  the  high  cost  of  collecting  data  and  maintaining  the  databases 
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which  can  benefit  other  programs  in  the  future.  There  is  no  focus  in  the  acquisition 
community  to  address  and  incentivize  PMs  to  address  the  external  benefits  of  their 
programs.  While  the  cost  of  performing  this  might  be  high,  or  perhaps  even  prohibitive 
to  one  project,  it  could  be  cost  effective  to  several  other  end  users.  “Institutional  bias 
forces  PMs  to  ignore  external  benefits.  A  PM  has  no  incentive  to  take  on  an  M&S 
investment  unless  he  can  justify  the  expense  from  his  existing  program.”  (Brown  et  al., 
2000)  Even  though,  according  to  DODI  7041.3,  societal  costs  and  benefits  outside  the 
Federal  Government  are  usually  not  included  in  a  DoD  Analysis,  it  is  pertinent  from  a 
business  perspective  to  consider  external  benefits. 

Perhaps  the  real  value  of  identifying  quantifiable  and  non- 
quantifiable  benefits  is  in  helping  others  outside  the  program  to  realize  potential 
synergies  of  reuse.  For  example,  the  Program  Executive  Officer  (PEO),  who  is  charged 
with  oversight  of  this  program  and  many  other  related  programs,  will  have  better 
visibility  into  requirements  and  the  potential  benefits.  He  can  more  accurately  assess 
M&S  investment  in  relation  to  a  broader  spectrum  of  programs.  Operational  analysis  and 
training  are  just  a  few  examples  of  synergies  of  reuse  between  acquisition  communities 
where  M&S  products  developed  for  testing  can  be  transferred  and  used  for  other  uses. 
Many  of  these  benefits,  while  external  to  an  individual  PM,  may  be  internal  benefit  from 
the  PEO’s  perspective.  The  PEO  must  be  provided  with  the  data  and  information  drawn 
from  the  business  case  analysis  performed  by  an  individual  program  under  his  purview  in 
order  to  make  sound  management  decisions.  Based  on  the  information  provided  in  the 
business  case  analysis,  the  PEO  can  choose  to  redirect  funding  from  other  sources  into 
the  program,  and/or  direct  a  PM  to  take  a  course  of  action.  This  PEO  action  may  not  be 
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cost-effective  in  a  micro-perspective  for  an  individual  program;  but  will  bring  an 
aggregate  gain  that  far  outweighs  the  individual  investment.  (Brown  et  al,  2000) 

4.  Step  4.  Evaluate  Alternatives 

To  evaluate  the  alternatives,  the  costs  and  benefits  of  each  alternative  must  be 
compared  and  then  ranked.  Such  comparisons  must  be  accomplished  using  both 
quantitative  and  qualitative  techniques  and  criteria.  Quantitative  techniques  include  net 
present  value,  benefit  cost  ratio,  return  on  investment,  payback  method,  internal  rate  of 
return,  hurdle  rate,  and  cost  effectiveness  analysis.  Qualitative  evaluation  considerations 
such  as  relationship  to  business  strategy,  schedule  risk,  organizational  and  technical  risks, 
social  benefits,  and  legal  and  regulatory  requirements  may  greatly  alter  the  quantitative 
ranking. 

The  choice  of  appropriate  tools  is  program  and  situation  dependent,  and  can 
greatly  influence  the  outcome  of  the  analysis.  These  tools  vdll  aid  decision-makers  in 
accurately  evaluating  all  alternatives  such  that  all  costs  and  benefits  are  viewed  on  a  level 
playing  field.  In  general,  each  feasible  alternative,  life-cycle  costs  and  benefits  are 
adjusted  using  discount  factors  to  accoxint  for  the  time  value  of  money.  A  complete 
analysis  properly  relates  quantitative  and  qualitative  factors.  Given  the  importance  of 
these  choices,  one  should  seek  expert  advice  and  guidance  before  proceeding. 

One  such  tool  is  the  Cost  Analysis  Strategy  Tool  (CASA).  CASA  is  ideal  for 
conducting  life  cycle  cost  estimates  to  include  supportability-related  trade-off  analyses, 
sensitivity  analyses  and  comparing  different  systems  and  alternative  support  structures. 

The  CASA  model  is  a  life  cycle  cost  decision  support  tool  for  Program  Managers 
responsible  for  materiel  acquisition  systems;  however,  it  can  be  reasonably  used  for  the 
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acquisition  of  models  and  simulations.  In  particular,  CASA  addresses  the  Total 
Ownership  Cost  (TOC)  for  the  objective  system  including  research,  development,  test, 
and  evaluation;  manufacturing  development  and  production;  and  the  entire  operational 
life  during  which  the  system  must  be  supported.  Virtually,  every  cost  associated  with  a 
system  is  covered  by  CASA,  whether  one-time,  recurring,  or  annual. 

The  CASA  model  has  extensive  analytical  capabilities.  In  addition  to  calculating 
life  cycle  cost  estimates  and  identifying  cost  drivers,  CASA  also  performs  trade-off  and 
sensitivity  analyses.  A  wide  range  of  sensitivity  analyses  can  be  conducted  on  the 
various  cost  parameters  included  within  the  CASA  model.  With  this  capability,  the  user 
can  examine  the  cost  impact  of  varying  factors  such  as  support  equipment  availability  or 
the  tumaroimd  time  for  spare  parts.  The  production  rate  and  quantity  buy  analysis  option 
assists  users  in  determining  the  optimum  quantity  of  times  to  procure.  The  robust  CASA 
life  cycle  cost  model  can  consider  life-cycle  studies  for  projects  that  last  up  to  50  years 
and  accommodate  customized  maintenance  schemes  with  up  to  10  levels.  CASA  is  a 
powerful  tool  for  developing  life  cycle  cost  estimates  and  gaining  a  better  understanding 
of  the  resultant  cost  figures  through  trade-off  and  sensitivity  analyses.  CASA  2000  is 
available  online  at  http://www.logsa.army.mil/alc/casa.  (McPherson,  2001) 

a.  Quantitative  and  Non-Quantitative  Evaluation  Methods 

Simulation  investment  alternatives  should  be  evaluated  using  multiple 
decision  attributes  that  include  both  financial  and  non-financial  criteria.  The  system  or 
process  for  analyzing  costs  and  benefits  associated  with  and  investment  should  include 
qualitative  and  quantitative  criteria  of  a  financial  and  non-financial  nature. 
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(1)  Quantitative  Methods.  This  section  addresses  the 
quantitative  methods  of  estimating  and  comparing  costs  and  benefits  of  different 
alternatives.  This  section  will  refer  often  to  entries  developed  in  Figure  1.  Benefit-Cost 
Calculations  in  Step  3  of  the  business  case.  Figure  1  provides  a  model  and  worksheet 
that  can  be  used  to  perform  the  quantitative  analysis  and  calculations  of  the  benefit  cost 
analysis.  This  section  will  provide  the  quantitative  analysis  methodologies. 

(2)  Net  Present  Value  (NPV).  Per  0MB  Circular  A-94,  NPV, 
the  discounted  monetized  value  of  expected  net  benefits,  is  the  standard  criterion  for 
deciding  whether  a  Government  program  can  be  justified  on  economic  principles.  Net 
present  value  is  calculated  by  assigning  monetary  values  to  benefits  and  costs, 
discounting  future  benefits  and  costs  using  an  appropriate  discount  rate,  and  subtracting 
the  sum  total  of  discounted  costs  from  the  sum  total  of  discounted  benefits.  Discoimt 
rates  and  the  discount  factors  to  be  used  are  provided  and  defined  in  0MB  Circular  A-94. 

Present  value  analysis  is  based  on  the  principle  that  benefits 
accruing  in  the  future  are  worth  less  than  the  same  level  of  benefits  that  accrue  now  and 
costs  that  occur  in  the  future  are  less  burdensome  than  the  costs  that  occur  now.  The 
formula  for  calculating  NPV  is:  NPV  =  Present  Value  of  benefits  -  Present  Value  (PV)  of 
costs. 

The  present  value  of  a  benefit  or  cost  is  calculated  by  multiplying 
the  amoimt  by  a  discoimt  factor.  The  discount  factor  is  equal  to  l/(l+i)n  where  i  is  the 
discoimt  rate  and  n  is  the  number  of  periods  over  which  discounting  takes  place.  If  the 
NPV  is  positive,  the  financial  return  on  the  project  is  economically  acceptable.  This  is 
because  the  cash  flows  generated  by  the  investment  over  its  economic  life  will:  recover 
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the  original  outlay  and  any  future  outlays,  earn  the  desired  return  on  the  outstanding 
balance,  and  provide  a  cushion  of  excess  economic  value.  If  the  NPV  is  negative,  the 
project  is  not  acceptable  on  economic  grounds. 

Additional  methods  of  evaluating  costs  and  benefits  are  available 
to  help  distinguish  among  alternatives  with  similar  NPVs  or  ones  where  it  is  difficult  to 
estimate  present  value.  0MB  Circular  A-94  recommends  that  when  a  net  present  value 
cannot  be  calculated,  agencies  provide  a  comprehensive  enumeration  of  the  different 
types  of  benefits  and  costs  and/or  quantify  benefits  and  costs  even  though  it  may  not  be 
possible  to  monetize  them. 

(3)  The  Benefit  Cost  Ratio  (BCR)  or  Profitability  Ratio.  The 
BCR  measures  the  economic  desirability  of  an  investment  by  dividing  the  present  value 
of  its  benefits  (cash  inflows)  by  the  present  value  of  the  costs  (outflows).  The  alternative 
with  the  highest  BCR  is  the  most  cost  effective  because  it  returns  the  most  benefits  per 
dollar  spent.  The  formula  for  this  ratio  is:  BCR  =  PV  (benefits)/  PV  (costs).  The  BCR 
provides  a  measure  of  the  benefits  obtained  per  dollar  spent.  The  higher  the  BCR  the 
larger  the  return.  Whereas  the  NPV  is  an  absolute  measure  that  refers  to  a  specific  set  of 
values,  the  BCR  allows  comparison  of  different  projects.  In  selecting  among  alternatives 
the  BCR  shows  which  alternative  provides  the  largest  return  relative  to  costs. 

(4)  ReUam  on  Investment  (ROI).  The  ROI  ratio  is  calculated 
by  dividing  the  average  armual  operating  cash  inflow  (benefit)  by  the  annual  net 
investment.  All  this  ratio  does  is  calculate  what  percentage  of  the  investment  the  annual 
benefit  cash  flow  is.  This  amount  is  calculated  on  an  annual  basis  and  the  formula  for 
this  ratio  may  be  calculated  as:  ROI  =  Average  annual  operating  cash  inflow  /  Net 


investment.  This  ratio  is  limited  in  its  usefulness  because  it  does  not  take  into 


consideration  the  economic  life  of  the  project  and  assumes  a  constant  annual  return.  The 
ratio  also  ignores  the  time  value  of  money  and,  therefore  should  not  be  used  except  in 
relation  to  annual  returns  or  when  speaking  in  generalized  or  gross  terms. 

(5)  Payback  Method.  This  method  estimates  the  time  it  takes 
to  recover  the  original  investment  outlay.  This  value  is  calculated  by  dividing  the  net 
investment  by  the  average  aimual  operating  cash  inflow:  Payback  (time)  =  Net 
investment  /  Average  annual  operating  cash  inflow.  This  ratio  gives  a  very  rough  test  as 
to  whether  the  investment  will  be  recovered  within  its  economic  life  span,  however  this 
ratio  is  limited  in  its  use  as  it  is  insensitive  to  the  economic  life  span  and  assumes 
constant  annual  operating  cash  inflows.  It  does  not  consider  cash  flows  beyond  payback 
and,  therefore,  does  not  measure  profitability. 

(6)  Internal  Rate  of  Return  (IRR).  The  IRR  method  determines 
the  discount  rate  that  makes  the  net  present  value  of  a  project  equal  to  zero.  When 
applied  to  both  cash  inflows  and  cash  outflows  over  the  project’s  economic  life,  it 
provides  a  zero  net  present  value  so  that  the  present  value  of  the  inflows  exactly  equals 
the  present  value  of  the  outflows. 

(7)  Hurdle  Rate.  The  hurdle  rate  is  a  minimum  standard  for  the 
return  required  of  an  investment.  A  hurdle  rate  may  be  used  to  help  select  from  among 
alternative  investments  when  other  decision  criterion  is  lacking.  For  instance,  a  hiudle 
rate  equal  to  the  cost  of  capital  as  reported  by  the  Treasury  Department  may  be  used. 

(8)  Cost  Effectiveness  Analysis.  0MB  Circular  A-94  states 
that  cost-effectiveness  analysis  is  appropriate  wherever  it  is  tinnecessary  or  impractical  to 
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consider  the  dollar  value  of  the  benefits  provided  by  the  alternatives  under  consideration. 
A  program  is  cost  effective  if,  on  the  basis  of  life  cycle  cost  analysis  of  competing 
alternatives,  it  is  determined  to  have  the  lowest  costs  expressed  in  present  value  terms  for 
a  given  amount  of  benefits.  Cost-effectiveness  analysis  is  appropriate  whenever:  (1)  each 
alternative  has  the  same  annual  benefits  expressed  in  monetary  terms  or  (2)  each 
alternative  has  the  same  annual  effects,  but  dollar  values  cannot  be  assigned  to  their 
benefits.  Cost-effectiveness  analysis  can  also  be  used  to  compare  programs  with 
identical  costs  but  differing  benefits.  In  this  case,  the  decision  criterion  is  the  discounted 
present  value  of  benefits. 

b.  Non-Quantitative  Evaluation  Considerations 

A  non-quantitative  evaluation  approach  often  has  to  be  used  when  there 
are  limited  measures  or  metrics  available  for  a  proposed  simulation  approach. 
Qualitative  evaluation  considerations  including  non-quantifiable  or  monetizable  benefits 
may  override  quantitative  criteria  in  the  ranking  or  acceptance  of  projects.  Such 
considerations  include:  relationship  to  business  strategy,  schedule  risk,  organizational  and 
technical  risks,  social  benefits,  or  legal/regulatory  requirements.  Non-quantifiable 
considerations  for  evaluating  alternatives  should  be  identified  in  the  BCA. 

(1)  Identifying  and  Evaluating  Risks.  Benefit  and  cost 

estimates  are  typically  uncertain.  Having  a  strategy  to  deal  with  the  risk  that  is  inherent 

in  large  simulation  investments/projects  is  critical.  One  of  the  greatest  risk  factors  to  the 

success  of  simulation  projects  is  the  amount  of  development  that  is  planned.  Full-scale 

development  is  where  the  potential  is  greatest  for  significant  cost  and  schedule  overruns 

and  lowered  performance  goals.  The  types  of  risks  encountered  in  a  simulation  project 

may  include:  schedule  risk,  risk  of  technical  obsolescence,  cost  risk,  technical  feasibility, 
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dependencies  between  a  new  project  and  other  projects  or  systems,  and  the  risk  of 
creating  a  monopoly  for  future  procurements. 

(2)  Risk  Management.  Risk  management  is  an  organized 
method  of  identifying  and  measuring  risk  and  developing,  selecting,  and  managing 
options  for  handling  risks.  Risk  management  consists  of  four  elements:  risk  assessment, 
risk  analysis,  risk  treatment  and  risk  management  plan.  Risk  assessment  which  identifies 
and  assesses  all  potential  risk  areas,  any  parts  of  a  project  where  there  is  uncertainty 
regarding  future  events  that  could  have  detrimental  effect  on  meeting  the  program  goal. 
Risk  assessment  continues  throughout  the  life  of  the  project  as  previous  uncertainties 
become  known  and  new  ones  arise.  Risk  analysis  characterizes  each  risk  as  to  the 
likelihood  of  its  occurrence  and  the  severity  of  its  impact.  It  results  in  a  watch  list  of 
potential  areas  of  risk.  Risk  analysis  also  continues  throughout  the  life  of  the  project. 
Risk  treatment  determination  is  made  after  risk  has  been  assessed  and  analyzed.  During 
risk  treatment,  a  determination  is  made  for  how  to  deal  with  it.  Alternatives  include: 

•  Transfer  -  risk  may  be  transferred  to  another  program  or  phase  of  the 
program,  another  Government  service  or  a  contractor. 

•  Avoidance  -  it  may  be  determined  that  the  risks  of  any  particular 
solution/altemative  are  too  great  and  the  alternative  should  be  removed 
form  further  consideration. 

•  Reduction  -  necessary  measures  can  be  identified  to  minimize  the 
likelihood  of  a  risk  occurring  and/or  minimize  the  damage  of  its  impact  on 
program  goals  should  it  occur. 

•  Assumption  -  a  decision  may  be  made  to  assume  a  risk  if  effective  control 
can  be  exercised,  the  probability  of  risk  is  small,  or  the  potential  damage  is 
either  minimal  or  too  great  for  another  program  or  Government  service  to 
bear. 

•  Sharing  -  if  a  risk  cannot  be  appropriately  transferred  and  should  not  be 
assumed,  it  can  be  shared  with  another  program  or  Government  service,  or 
a  contractor. 
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(3)  Risk  Management  Plan.  A  risk  management  plan  should  be 
developed  that  includes  information  on  the  types,  probability  and  impact  of  risks 
pertinent  to  the  simulation  project,  including  the  risk  that  the  funding  request  will  not  be 
approved  or  not  approved  in  its  entirety  and  plans  for  how  to  treat  and  manage  the  risk,  to 
include  how  to  respond  to  lower  funding.  Furthermore,  requiring  a  higher  return  for 
projects  determined  to  be  of  higher  risk  can  accommodate  risk.  Also,  risk  analysis 
estimates  of  the  probability  that  a  simulation  investment  will  fail  and  the  impact  this 
would  have  on  the  business  can  be  subtracted  from  the  expected  benefits  to  adjust  the 
ROI  or  NPV  calculations  to  reflect  risk. 

Sophisticated  risk  assessment  methodologies,  such  as,  probabilistic 
simulation  can  be  used  to  estimate  ranges  for  total  annual  cash  flows  and  key  variables 
can  be  identified.  Probability  distributions  can  then  be  assigned  to  the  outcomes  for  each 
of  the  variables.  Computers  can  be  used  to  run  multiple  iterations.  A  contractor  can 
perform  an  independent  risk  analysis  of  the  selected  approach  or  alternative  and  the  ROI 
can  be  adjusted  accordingly. 

5.  Step  5.  Conduct  Sensitivity  Analysis 

Sensitivity  analysis  is  an  essential  step  in  the  decision  process  as  it  accoimts  for 
ever-present  imcertainties  and  variable  changes.  Such  analysis  repeats  the  evaluation  of 
alternatives  performed  in  Step  3  but  with  changes  to  the  xmcertain  variables  and  examines 
the  effect  on  the  final  decision.  The  outcome  of  the  sensitivity  analysis  will  provide  a 
better  understanding  of  the  robustness  of  the  output  of  the  business  case  analysis. 

Sensitivity  analysis  is  highly  recommended,  even  if  there  appear  to  be  significant 
differences  among  the  alternatives,  because  an  apparently  superior  solution  may  be  very 
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sensitive  to  changes  in  a  single  variable.  Sensitivity  analysis  is  required  when  differences 
among  alternatives  are  less  obvious  and  may  be  totally  driven  by  variability  of  key  input 
factors.  “The  key  factors  to  be  tested  may  include,  but  are  not  limited  to;  project  or 
program  length,  volume  or  quantity  and  mix  of  production  units,  requirements, 
configurations,  assumptions,  and  discount  rates  and  other  economic  factors.”  (Brown  et 
al.,  2000) 

To  conduct  the  sensitivity  analysis,  all  parameters  in  the  analysis  are  held  constant 
except  the  factor  being  tested.  The  analysis  is  then  reworked  using  different  estimates  for 
the  factor  under  review.  If  this  results  in  changes  to  the  ranking  of  alternatives,  the 
analysis  is  sensitive  to  that  amount  of  change  in  the  variable.  Each  parameter  should  be 
tested  individually  to  determine  its  effect  on  the  analysis. 

Sensitivity  refers  to  the  relative  magnitude  of  change  in  one  or  more  elements  of 
an  economic  analysis  that  will  cause  a  change  in  the  ranking  of  alternatives.  Sensitivity 
analysis  is  used  for  assessing  the  extent  to  which  costs  and  benefits  are  sensitive  to 
changes  in  key  factors.  In  a  sensitivity  analysis,  if  one  particular  factor  or  cost  element 
can  be  varied  over  a  wide  range  without  affecting  the  ranking  of  alternatives,  the  analysis 
is  said  to  be  insensitive  to  uncertainties  regarding  the  particular  event.  A  sensitivity 
analysis  can  provide  a  range  of  costs  and  benefits  that  are  likely  to  be  a  better  guide  then 
a  single  estimate. 

If  there  is  certainty  and  the  preference  ranking  establishes  one  alternative  as 
markedly  superior  to  the  rest,  sensitivity  analysis  is  probably  lumecessary.  However  if 
there  is  uncertainty  with  at  least  some  of  the  assumptions  and  the  alternative  of  choice  is 
not  clearly  preferable  to  the  rest,  then  a  sensitivity  analysis  may  be  necessary. 
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As  part  of  the  sensitivity  analysis,  major  assumptions  should  be  varied  and  net 
present  value  and  other  outcomes  recomputed  to  determine  how  sensitive  outcomes  are  to 
changes  in  the  assumptions.  Assumptions  deserving  the  most  attention  will  depend  on 
the  dominant  benefit  and  cost  elements  and  the  areas  of  greatest  uncertainty.  For  each 
alternative,  key  high  risk  factors  should  be  changed  to  a  less  favorable  munber  to  test 
sensitivity.  Key  elements  to  evaluate  include:  length  of  project  life;  volume,  mix,  or 
pattern  of  workload;  requirements;  configuration;  assumptions;  discount  rates;  and  cost 
and  benefit  estimates.  (GSA,  2000) 

6.  Step  6.  Developing  a  Migration  Strategy 

After  determining  the  best  alternative,  one  must  develop  a  sound  implementation 
plan  to  migrate  the  chosen  strategy  into  the  program  (Brown  et  al.,  2000).  Cost,  schedule 
and  performance  goals  must  be  formalized  in  order  to  manage  the  investment/project. 
The  plan  for  migration  must  incorporate  a  systematic  approach  to  ensure  the  developer 
implements  the  identified  drivers  and  captures  the  expected  benefits.  The  performance 
goals  should  stem  from  the  needs/requirements  that  alternatives  are  fulfilling  and  should 
address  the  benefits  they  are  expected  to  provide.  Performance  goals  should  be 
consistent  with  the  organization’s  strategic  plan  goals  and  should  be  linked  to  or  part  of 
performance  plan  goals  and  measures.  Schedule  and  cost  goals  must  also  be  established 
to  help  ensure  projects  adhere  to  plaimed  costs  and  schedules.  Interim  annual  goals  and 
measures  must  be  established  for  multi-year  projects  to  ensiue  timely  detection  of 
problems  and  implementation  of  corrective  action.  Implementation  of  the  migration 
strategy  will  undoubtedly  force  changes  to  the  program’s  plan  and  budget.  If  a  new  tool 
or  process  is  expected  to  save  money,  then  those  savings  should  be  subtracted  from  that 

part  of  the  program  budget  and  reassigned  elsewhere  as  an  up-front  action. 
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7.  Step  7.  Monitoring  the  Process  and  Assessing  Results  through 
Formalized  Feedback 

The  final  step  in  developing  a  business  case  will  be  to  create  metrics  to  assess 
progress  towards  the  overall  vision.  These  metrics  should  be  tied  to  the  changes  made  to 
the  program’s  acquisition  plan,  to  provide  timely  feedback  on  their  success  in  meeting 
desired  results  in  performance,  schedule,  and  cost.  Performance  metrics  should  stem 
fi'om  the  needs  and  requirements  that  alternatives  are  fulfilling,  and  should  address  the 
benefits  they  are  expected  to  provide.  Schedule  and  cost  metrics  must  also  be  developed 
to  help  ensure  programs  adhere  to  planned  costs  and  schedules  (Kidwell,  1998;  GSA, 
1998). 

Since  PMs  face  tremendous  pressure  to  bring  a  product  into  use,  they  must  deliver 
their  programs  with  complementary  benefits  first.  This  is  their  true  priority,  but  they 
should  also  identify  real  or  potential  external  benefits  up  the  management  chain  to  the 
PEO.  That  office  can  then  make  more  informed  decisions  on  the  macro  benefits.  PMs 
should  consider  increasing  investment  earlier  in  the  program  if  the  business  case  strongly 
indicates  downstream  savings  as  a  result.  PEOs  can  provide  the  attendant  oversight  and 
direction,  with  a  requisite  re-allocation  of  funds  when  it  is  in  the  DoD’s  best  interest  to  do 
so.  (Brown  et  al.,  2000) 

Monitoring  needs  to  be  a  truly  integrated  process,  with  all  elements  actively 
involved.  A  sharing  of  analysis,  combined  with  a  DoD  commitment  to  maximize  and 
optimize  any  potential  benefits  of  M&S  technology,  will  bring  unprecedented  reward  in 
cheaper,  better,  stronger  products  and  the  associated  prudence  in  managing  the  public 
resources.  Conducting  a  Post  Implementation  Review  that  includes  financial  data,  using 
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measures  and  metrics  developed  enables  the  monitoring;  feedback  and  continuous 
improvement  of  the  M&S  product  and  development  process. 

a.  Conduct  Post  Implementation  Review 

Once  a  system  becomes  folly  operational  or  implemented,  a  Post 
Implementation  Review  (PIR)  will  be  conducted.  This  review  should  occur  about  3  to  6 
months  after  the  project  has  become  operational.  It  is  highly  recommended  that  the 
review  be  conducted  by  a  group  other  than  the  IPX  that  has  been  responsible  for  the 
development  of  the  system.  This  ensures  that  it  is  conducted  independently  and 
objectively.  Subsequent  PIRs  should  be  conducted  on  a  periodic  basis  after  the  first  PIR 
to  ensure  that  the  completed  system  is  continuing  to  meet  organizational  and  user  needs. 

Each  PIR  that  is  conducted  has  a  dual  focus.  First,  it  provides  an 
implementation  assessment  of  the  system,  including  an  evaluation  of  the  development 
process.  Secondly,  it  indicates  the  extent  to  which  the  business  case  steps  are  sustaining 
or  improving  the  success  rate  of  M&S  projects.  The  following  areas  should  be  evaluated 
as  part  of  a  complete  PIR.  Each  topic  should  be  dociunented  with  a  summary  of  findings 
that  support  the  conclusions  and  recommendations.  Additional  Documentation  and 
Project  Risk  Checklists  are  included  in  Appendices  D  and  E. 

b.  Post  Implementation  Review  Assessments 

(1)  Mission.  An  analytical  approach  should  be  taken  to 

determine  whether  the  implemented  system  has  achieved  its  proposed  impact  on  the 

agency’s  business.  It  is  important  that  all  agency  M&S  investments  are  aligned  with  the 

organization’s  mission  and  the  agency’s  program  objectives.  Additionally,  M&S 

investments  are  to  be  analyzed  and  evaluated  in  respect  to  the  overall  benefits  for  DoD 

business  practices.  The  PIR  team  is  obligated  to  determine  the  status  of  several  project 
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variables,  including  the  delivery  of  services  or  products  estimation  of  cost  savings, 
compliance  with  the  DoD  High  Level  Architecture,  evaluations  of  the  information 
product,  e.g.  accuracy,  timeliness,  adequacy,  and  appropriateness  of  information, 
identification  of  additional  maintenance  and  security. 

(2)  Systems  Description.  A  specific  description  of  the 
functions  of  the  system  should  be  documented.  The  tasks  performed  and  the  approach 
taken  to  accomplish  each  task  and  the  resources  used  also  needs  to  be  outlined.  All 
hardware,  software,  and  applications  software  associated  with  the  system  should  be 
documented  as  well.  All  personnel  requirements  and  geographic  locations  that  provide 
input,  receive  output,  or  assist  in  system  processing  should  be  identified.  Finally,  there 
should  be  an  explanation  of  how  the  system  contributes  to  the  organization’s  mission. 

(3)  Change  Control.  The  Change  Control  process  and 
procedures  for  the  system  should  be  documented  and  evaluated  for  efficiency.  A 
determination  should  be  made  of  the  number  and  severity  of  the  changes  to  date  and  their 
impact  on  the  stability  of  the  system.  An  assessment  should  also  be  made  describing  the 
system’s  ability  to  respond  to  changing  requirements. 

(4)  Operation.  An  analysis  of  the  system  operation,  including 
hardware,  system  and  application  software  should  be  conducted  and  compared  against 
those  projected.  Finally,  recommendations  regarding  system  changes  and  redesign  based 
on  projected  comparisons  and  operation  problems. 

(5)  Security.  A  security  evaluation  should  be  conducted  to 
verify  that  the  appropriate  security  requirements  are  documented  and  enforced.  If 
problems  are  identified  in  this  area,  these  should  be  outlined  and  corrective  actions  need 


59 


to  be  identified.  Any  security  or  risk  incidents  need  to  be  identified  and  analyzed  for 
potential  system  weaknesses.  An  evaluation  should  be  made  of  the  cost  effectiveness  of 
system  security  measures  and  recommendations  made  where  improvements  can  be  made. 
Finally,  the  contingency  plans  need  to  be  checked  to  ensure  that  they  are  current  and  are 
feasible  to  minimize  loss  from  threats  and  equipment/software  malfunctions. 

(6)  Outputs.  The  outputs  of  the  new  system,  e.g.  reports,  data, 
or  formats,  need  to  be  compared  to  those  that  were  initially  proposed.  The  impact  of  any 
changes  on  the  initial  design,  geographic  locations,  or  telecommunications  factors  should 
also  be  evaluated  and  documented. 

(7)  Documentation.  Any  system  documentation  such  as  User’s 
Guides  or  Operations  Manuals  should  be  reviewed  for  completeness,  accuracy,  and 
timeliness.  A  list  of  all  required  documentation  should  be  developed  and  kept  up  to  date. 

(8)  Management.  A  review  of  the  support  organization 
structure  should  be  examined.  The  organizational  structure  and  responsibilities  as 
implemented  should  be  compared  against  those  documented  during  the  project.  The 
system  ownership  and  individual  authorities  and  responsibilities  should  be  verified  and 
updated,  as  required.  Any  areas  where  there  is  conflicting,  unidentifiable  or 
inappropriate  management  or  supervision  should  be  identified  and  corrected.  Training 
issues  should  also  be  examined  in  this  area  to  ensure  that  personnel  (users  and  support) 
are  properly  trained.  (GSA,  2000) 

c.  Review  and  Evaluate  Project  Information 

In  addition  to  the  items  reviewed  in  the  Post  Implementation  Review,  the 
following  areas  need  to  be  evaluated  and  assessed: 


60 


(1)  Technical  Capability.  The  technical  capabilities  of  the 
project,  both  current  and  future,  should  be  reviewed  and  evaluated.  Factors  such  as  the 
competency  of  the  workforce  to  use  the  new  system  and  employee  satisfaction  or 
retention,  the  extent  to  which  advanced  technology  was  used,  and  the  methodological 
expertise  of  the  development  team  should  be  considered. 

(2)  Measurements  of  Actual  vs.  Projected  Performance.  The 
project’s  actual  results  should  be  compared  to  planned  estimates  in  terms  of  cost, 
schedule,  performance,  and  mission  improvement  outcomes.  An  attempt  should  also  be 
made  to  determine  the  causes  of  major  differences  between  the  planned  and  final  results. 

(3)  Evaluation  of  Outstanding  Issues.  If  the  PIR  reveals  issues 
that  still  require  attention,  these  issues  need  to  be  identified  and  documented.  The  issues 
should  clearly  document  the  estimates  of  cost  and  time,  the  risks  for  not  addressing  the 
issue,  any  tradeoffs  or  alternatives,  and  provide  a  recommendation  from  the  PIR  review 
team.  The  issues  should  then  be  sent  to  senior  management  for  evaluation  and  a  final 
decision  on  the  actions  to  be  taken. 

d.  Review  and  Update  Financial  Information  &  Performance 

Measures 

Once  the  actual  final  financial  and  performance  measurement  information 
for  the  initiative  has  been  gathered,  it  should  be  compared  against  the  planned  results. 
This  will  allow  a  determination  to  be  made  as  to  the  success  of  the  initiative  and  to 
determine  the  causes  of  any  differences  between  planned  and  actual  results.  The 
following  five  areas  should  be  evaluated  in  this  process: 

•  Evaluation  of  Cost  Information  -  compares  the  actual  versus  planned  life 
cycle  costs  for  the  initiative. 
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•  Evaluation  of  Financial  Return  Information  -  compares  the  actual  versus 
planned  results  for  financial  performance  measures. 

•  Evaluation  of  Non-Financial  Return  Information  -  compares  the  actual 
versus  plarmed  results  for  non-financial  performance  measures. 

•  Evaluation  of  Acquisition  and  Procurement  Information  -  compares  the 
actual  versus  plarmed  results  for  contract  and  contractor  information. 

•  Evaluation  of  Budget  and  Financing  Information  -  compares  the  actual 
versus  planned  results  for  funding  source  and  general  budget  and 
financing  information. 

The  monitoring  and  feedback  step  continues  throughout  the  development 
and  post  deployment  of  the  system.  The  collection  of  system,  process  and  financial 
information  creates  a  M&S  development  and  deployment  which  can  be  audited  and 
additionally,  helps  the  PM  achieve  his  programmatic  goals. 
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IV.  APPLYING  THE  BUSINESS  CASE  FRAMEWORK  TO 
DEVELOPMENTAL  TESTING 


A.  INTRODUCTION 

This  part  of  the  thesis  will  take  the  business  case  developed  in  the  previous 
section  for  programs  in  general  and  apply  it  to  Test  and  Evaluation  (T&E).  The 
remainder  of  the  thesis  will  deal  primarily  with  the  discipline  of  Developmental  Testing 
(DT).  T&E  are  essential  parts  of  the  development  and  deployment  of  all  Army  systems. 
“The  information  generated  as  a  result  of  T&E  influences  every  action  taken  during  the 
system  acquisition  process.”  (HQDA,  1997)  The  Developmental  Test  Command  (DTC), 
a  subordinate  command  of  the  Army  Test  and  Evaluation  Command  (ATEC),  has  the 
roles  of  developmental  testing  and  safety  certification,  and  advises  program  managers  as 
part  of  IPTs. 

DTC  has  a  long  history  of  using  physical  simulation  as  part  of  the  process  to 

confirm  a  developmental  weapon  system’s  readiness  and  technical  maturity.  M&S 

within  the  T&E  process  is  the  integration  of  a  mix  of  computer  simulation,  actual 

warfighting  systems,  and  weapon  system  simulators,  inunersed  in  synthetic 

environments,  distributed  geographically,  and  connected  through  high-speed  networks. 

Simulation  for  T&E  includes  Software-in-the-Loop  (SWIL),  Hardware-in-the-Loop 

(HWIL),  or  Human-in-the-Loop  (HITL)  simulations  as  well  as  synthetic  environment 

simulations.  T&E  uses  M&S  to  identify  test  parameters  and  drivers  for  field  tests; 

determine  high  risk  areas;  predict  test  results;  assist  in  the  allocation  of  scarce  test 

resources;  and  provide  entity  stimulation  in  support  of  interoperability  testing.  M&S 

provide  the  only  route  to  obtain  risk-reducing  system-performance  data  in  situations  that 
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cannot  be  tested  due  to  safety,  cost,  or  other  constraints.  Presently,  DTC  is  reengineering 
the  Army’s  technical  test  capabilities  through  its  VPG  initiative.  The  VPG  consists  of  a 
comprehensive  and  interrelated  set  of  synthetic  environments,  stimulators,  and  simulation 
test  procedures  operating  within  a  standard  architecture  framework.  Historical  data  from 
hardware  tests  on  DTC  ranges  and  other  real-world  test  sites  provide  ground  truth  data 
that  anchor  the  VPG  to  reality. 

DTC  seeks  a  sound  business  methodology  to  provide  cost  avoidance  and  Return 
on  Investment  (ROI)  data  for  future  tests  using  M&S.  DTC  needs  the  data  to  ensure  that 
it  can  recommend  the  right  mix  of  live  testing,  which  validates  the  M&S,  and  use  of  the 
VPG  or  virtual  testing,  that  augments  the  live  testing.  The  cost  data  help  DTC  encourage 
investment  in  its  live  and  VPG  facilities  and  technologies.  “M&S  is  useful  for  aiding  in 
test  design  and  making  pre-test  predictions.  Test  results  are  invaluable  in  validating  and 
improving  models  and  simulations”  (Coyle,  O’Bryon  &  Hillegas,  2000).  These  data  will 
be  most  credible  to  customers  if  developed  as  applications  of  Government  approved 
methodologies.  DTC  therefore  seeks  to  ensure  that  its  VPG  cost  estimation 
methodologies  are  consistent  with  Government  guidance  on  the  role  of  T&E  in  system 
acquisition  (HQDA,  1997),  on  cost  estimation  (CEAC,  1997),  and  on  economic  analysis 
(CEAC,  1995).  This  chapter  will  show  how  the  business  case  for  M&S,  presented  in 
Chapter  3,  can  accomplish  this  goal.  The  business  case  approach  can  help  illuminate  the 
point  that  “M&S  can  be  more  effective  and  efficient  when  the  program  office  develops  a 
formal  plan  early  in  the  program  and  secures  a  ‘buy-in’  to  the  plan  by  the  acquisition 
decision-makers  and  the  testing  community”  (Coyle,  O’Bryon  &  Hillegas,  2000). 
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DTC  sponsored  studies  to  assess  the  benefits  of  using  virtual  testing  techniques  in 
three  historical  programs.  The  PMs  of  the  weapon  system  acquisition  programs  would 
ultimately  realize  these  benefits.  The  historical  programs  were  the  Enhanced  Position 
Location  Reporting  System  (EPLRS)  (Brugh,  1997),  the  Automatic  Chemical  Agent 
Detector  Alarm  (AC  AD  A)  (Brugh,  1997),  and  the  Abrams  M1A2  Tank  (Brugh,  1996) 
technical  test  programs.  Thefocus  of  the  studies  was  to  estimate  the  actual  and  potential 
savings  to  cost  and  schedule  that  were  achieved  through  the  use  of  virtual  test  techniques 
when  compared  to  using  live  test  techniques..  Another  DTC  study  estimated  cost 
avoidance  at  the  Simulation/Test  Acceptance  Facility  (STAF)  (Johnson,  1996).  It  is  a 
goal  of  this  thesis  to  apply  the  business  case  methodology  developed  in  Chapter  3  to 
these  previous  studies  to  validate  the  need  for  the  use  of  such  a  business  case  in  all  major 
decisions  involving  the  use  of  M&S  in  developmental  testing. 

B.  BUSINESS  CASE  APPLIED  TO  DT  CASES 

1.  Step  1.  Baseline 

Since  the  key  in  determining  the  success  of  the  selected  programs  lies  in 
establishing  their  initial  state  before  any  changes  in  procedure  were  considered  or 
implemented,  the  performance  baseline  measurements  for  the  EPLRS,  ACAD  A,  M1A2 
and  the  STAF  will  be  addressed  herein.  This  will  be  done  m  order  to  show  the 
changes/improvements  the  programs  vmderwent  during  the  different  testing  events.  The 
baseline  measures  were  taken  using  live  testing  techniques  without  the  use  of  formal 
M&S  techniques. 

The  baseline  objective  was  to  compare  the  cost/schedule  of  the  virtual  testing  with 

the  same  amount  of  testing  using  live  techniques.  Another  major  metric  used  for  the 

studies  was  the  “Number  of  areas  assessed  that  are  difficult/impossible  to  test  physically 
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due  to  limitations  in  cost,  time,  manpower,  or  due  to  risk  to  humans,  equipment,  or  the 
environment.”  (Patenaude,  1996)  This  metric  was  used  for  all  of  the  studied  programs 
and  was  the  main  reason  for  the  comparison  between  EPLRS  I/II  and  EPLRS  III.  Cost 
and  time  would  be  prohibitive  if  human  radio  operators  were  used  to  obtain  the  required 
data.  Other  examples  of  this  metric  are  Electromagnetic  Interference,  Nuclear 
Survivability,  and  environmental  testing  (Brugh,  1997).  An  additional  metric  was 
“Quality  and  quantity  of  test  data.”  (Patenaude,  1 996)  As  might  be  expected,  open  air, 
live  field  testing  with  chemical  agents  is  not  practical/feasible.  Chemical  agent  detector 
testing  in  the  past  has  essentially  always  been  conducted  in  environmentally  controlled 
chambers;  therefore,  the  past  methodology  for  testing  involving  chemical  agents  was 
actually  a  “synthetic  environment”  in  the  broadest  sense.  The  comparison  was  made 
between  the  cost/schedule  during  the  ACADA  Production  Verification  Test  (PVT)  using 
the  Detector  Test  System  (DTS)  and  the  cost/schedule  if  the  tests  were  conducted  in  a 
standard  chamber.  Since,  the  baseline  should  include  a  clear  emmciation  of  assumptions 
and  constraints,  it  should  be  understood  that  this  is  very  difficult  in  the  changing 
environment  of  M&S  where  techniques  and  technology  opportunities  are  changing 
almost  weekly.  The  main  metric  used  was  the  “Number  of  areas  assessed  that  are 
difficult/impossible  to  test  physically  due  to  limitations  in  cost,  time,  manpower,  or  due 
to  risk  to  humans,  equipment,  or  the  environment.”  (Patenaude,  1996) 

2.  Step  2.  Vision/Direction 

Current  force  reductions  have  lead  to  restructuring.  DTC,  in  order  to  continue  its 

testing  mission  effectively,  must  downsize  and  restructure  to  test  smarter  with  fewer 

resources.  Current  and  nascent  technologies  are  the  ideal  means  for  accomplishing  this 

goal.  DTC  has  a  long  history  of  using  physical  simulation  to  confirm  a  weapon  system’s 
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readiness  and  technical  maturity;  presently  DTC  is  reengineering  the  Army's  test 
capability  through  its  VPG  initiative.  The  VPG  consists  of  a  comprehensive  and 
interrelated  set  of  synthetic  environments,  stimulators,  and  simulation  test  procedures 
operating  within  a  standard  architectural  framework.  The  VPG  will  be  used  to  confirm  a 
system's  readiness  and  technical  maturity,  from  concept  through  fielding,  and  with  a 
substantial  reduction  in  program  resources.  Historical  data  from  actual  hardware  tests  on 
DTC  ranges  and  other  real-world  test  sites  provide  groimd  truth  data  that  anchor  the  VPG 
to  reality. 

In  order  to  prove  the  need  to  invest  in  new  testing  technology,  DTC  must  prove 
the  worth  of  developing  a  VPG  on  a  continuous  basis.  The  ultimate  goals  of  VPG  are  to 
accomplish  testing  better,  faster  and  cheaper.  One  method  of  proving  the  worth  of  VPG 
is  through  cost  avoidance  and  benefit  analyses.  A  cost  avoidance  methodology  for  VPG 
will  be  presented  herein.  Several  studies  have  been  conducted  to  show  the  results  of 
some  early  VPG  investments  that  benefited  the  testing  community  and  the  PM  ultimately. 
The  results  of  these  studies  will  be  used  to  demonstrate  the  validity  of  the  cost  avoidance 
model  and  the  ROI. 

Most  of  the  testers  who  discuss  this  subject  agree  that  a  VPG  runs  much  the  same 
as  a  physical  proving  ground,  except  that  the  nature  of  the  item  being  tested,  the 
mechanisms  of  communication,  data  collection  methods,  and  other  basic  functions  shift 
toward  the  digital  computer  model.  DTC  has  tested  physical  hardware  and  prototypes  on 
ranges  and  in  laboratories  for  decades  (physical  testing);  the  VPG  extends  DTC’s  test 
capability  into  the  domain  of  models,  virtual  prototypes,  and  hardware-in-the-loop 
simulations  (virtual  testing). 
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Three  testing  paradigms,  physical,  and  virtual,  help  one  to  distinguish  among  the 
paradigms  and  explain  the  realm  of  the  virtual  proving  ground.  Physical  testing  is  a 
hardware  process  that  yields  original  knowledge  and  experience  pertaining  to  the 
functionality  of  the  weapon  system,  subsystems,  and  components  as  well  as  the  nature 
and  effects  of  the  natural  and  man-made  environments  (Grormd  Truth).  Data  from 
physical  testing  serves  as  the  basis  for  functional  and  conceptual  models  of  both  the 
systems  and  the  environments.  Virtual  testing  is  comprised  of  a  mix  of  hardware 
systems,  subsystems,  components  and/or  prototypes;  soldier-in-the-loop  simulators;  and 
digital  models.  Data  collected  from  virtual  testing  is  based  on  the  system,  subsystem,  or 
component  performance  of  the  physical  hardware  or  human  in  the  loop  simulations 
coupled  with  the  modeled  effects  of  the  remainder  of  the  system  and  environment.  This 
virtual  test  data  contains  a  level  of  uncertainty  due  to  the  assumptions  and 
approximations  used  in  the  digital  models.  However,  the  application  of  a  rigorous 
verification,  validation  and  accreditation  process  can  minimize  this  uncertainty. 

Given  the  above  basic  imderstanding  of  the  realm  of  the  VPG,  it  is  reasonable  to 
ask  how  the  VPG  operates  and  who  the  players  are.  The  current  VPG  operational  players 
include  DTC  test  centers.  Operational  Test  Command  (OTC),  Research  and  Development 
Centers  (RDECs),  PMs,  and  contractors.  After  full  VPG  development,  the  players  will 
have  additional  testing  tools  in  the  constructive  simulation  arena  that  will  allow  testing 
that  is  no  longer  limited  to  physical  proving  grounds  or  test  centers.  For  some  completely 
constructive  testing,  the  concept  of  taking  a  piece  of  hardware  to  a  specific  physical 
location  will  no  longer  be  relevant.  DTC  will  still  be  the  tester  in  the  sense  that  DTC  test 
centers  will  produce  models  of  the  test  process,  and  ground  truth  data  needed  for  both 
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producing  and  validating  the  system  and  environment  models.  This  role  of  the  virtual 
tester  is  complementary  to  the  classic  physical  testing  mission  that  is  necessary  to 
continue  regardless  of  the  sophistication  of  the  emerging  VPG.  Important  players  are  the 
other  members  of  the  IPT  and  the  PMs.  It  is  reasonable  to  visualize  the  PM  office  for  a 
developmental  system  as  the  organization  that  links  to  various  players  to  obtain  the 
necessary  testing  models  and  other  pertinent  information  to  run  a  test  on  a  system  model 
developed  at  the  PM’s  organization.  For  instance,  running  such  a  test  involves  sharing 
the  system  model  and  its  associated  test  issues  and  criteria  with  the  DTC  agency,  which 
tailors  the  model  for  the  specific  test.  All  of  this  takes  place  within  a  distributed,  but 
coimected,  complex  of  organizations  that  contribute  to  the  virtual  testing  process. 

As  mentioned  above,  a  reasonable  place  to  start  an  analysis  of  the  VPG  process  is 
the  hardware  testing  process.  Figure  2  is  a  simplified  diagram  of  the  process  of 
conducting  hardware  tests  at  fixed  ranges  and  test  centers.  Basically,  the  process 
involves  a  physical  system  that  is  designed  and  manufactured  to  satisfy  a  set  of  critical 
issues  and  associated  criteria,  and  tested  to  those  issues  and  criteria.  A  set  of  test  plans  is 

derived  from  the  requirements  of  those  issues  and  criteria.  These  test  plans  contain  the 

♦ 

test  procedures  and  call  for  specific  instrumentation  to  obtain  test  data.  Test  resources 
and  historical  data  also  contribute  to  test  plans.  Tests  are  executed  from  the  plans.  Test 
outputs  are  in  the  form  of  test  data.  System  assessors  use  the  data  to  determine  how  well 
test  issues  are  satisfied  and  criteria  are  met.  Classically,  DTC  is  the  major  player  in 
essentially  all  Of  the  boxes  of  this  physical  hardware  technical  testing  process.  This  is 
due  to  the  fact  that  the  developer  sends  the  hardware  system  to  a  DTC  test  center  or 
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proving  groimd  to  be  tested.  Additional  weapon  system  developer  resources  or 
equipment  are  often  added  at  the  test  site  or  range  to  execute  tests  and  obtain  data. 


HARDWARE  TESTING  PROCESS 


Figure  2.  Hardware  Testing  Process.  (Source:  Researcher) 

The  virtual  testing  process.  Figure  3,  is  diagrammed  in  much  the  same  way  as 
Figure  2,  but  differs  in  two  basic  ways.  First,  systems  being  tested  and  test  procedures 
themselves  tend  to  be  in  digital  forms  rather,  than  physical  forms.  The  digital  form 
includes  the  virtual  testing  zone  where  humans  and  hardware  in  the  loop  simulations  are 
challenged  by  synthetic  stimuli.  Second,  Figure  3  follows  the  VPG  operational  concept 
that  distributes  the  responsibility  for  testing  to  those  organizations  that  are  a  part  of  the 
distributed  VPG  complex.  The  testing  process  is  on  the  centerline  of  Figure  3. 
Support/infrastructure  are  shown  above  and  below  the  centerline.  The  Intemal/Extemal 
Interface  Infrastructure  represents  principally  the  networked  electronic  infrastructxire  that 
links  players  in  the  VPG  testing  process. 
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3.  Step  3.  Quantify  the  Costs  and  Benejfits  of  Alternatives/Capabilities 

According  to  the  Army,  the  “Economic  Analysis  Process”  (CEAC,  1995)  is  an 

eight  step  process:  (1)  Establish  objective,  (2)  Formulate  assumptions,  (3)  Identify 

constraints,  (4)  Identify  alternatives,  (5)  Estimate  costs  and  benefits  for  each  alternative, 

(6)  Compare  alternatives,  (7)  Perform  sensitivity  analysis,  and  (8)  Report  results  and 

recommendation.  The  Department  of  the  Army  Economic  Analysis  Manual  recommends 

that  an  economic  analysis  study  plan  include  the  mission,  background,  purpose, 

constraints,  assumptions,  cost  element  stnxcture,  cost  and  benefit  estimating 

methodology,  system  description,  configuration,  schedules,  and  issues.  The  level  of 

economic  analysis  detail  increases  with  project  dollar  value  or  project  visibility.  Among 

the  limitations  with  economic  analysis  discussed  in  the  manual  are  that  economic  analysis 

cannot  be  applied  with  cookbook  precision,  but  must  be  tailored  to  fit  the  problem,  and 
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that  it  is  not  a  substitute  for  sound  judgment,  management,  or  control.  The  Business  Case 
Framework  outlined  in  detail  in  Chapter  3  of  this  thesis  has  all  of  the  steps  outlined  by 
the  Army  and  features  the  characteristics  necessary  to  make  a  business  decision  to  make  a 
decision  about  using  M&S  in  the  discipline  of  testing.  Therefore,  the  techniques  for 
economic  analysis  detailed  in  the  Business  Case  Framework  outlined  in  Chapter  3  of  this 
thesis  will  be  used  to  do  the  analysis  of  the  four  systems. 

a.  Cost  Analysis  Metrics 

“A  true  measurable  metric  would  be  any  numerical  value  that  enables  us 
to  assess  how  much  faster,  how  much  better,  and  how  much  cheaper  a  given  acquisition 
process  is  or  can  be  fielded.”  (Pinker,  1997)  Measurable  T&E  cost  metrics,  live  or  VPG, 
include  cost  of  consumable  items,  cost  of  facilities,  cost  of  technology,  and  test-personnel 
costs.  Others  are  schedule  with  consideration  for  acquisition  phase  time,  administrative 
lead-time,  and  of  course  test  time.  Developmental  T&E  affect  program  cost  through 
knowledge  of  performance  and  the  design  changes  necessary  to  implement  a  system  that 
does  not  meet  requirements.  Developmental  T&E  provide  benefits  in  troubleshooting 
and  data  requirements  generation. 

These  and  other  metrics  may  apply  to  cost  savings,  cost  avoidance  and 
productivity  improvements.  Cost  savings  result  in  the  reduction  of  cost  of  approved 
budget  items.  A  cost  avoidance  is  the  result  of  an  action  taken  in  the  immediate  time 
frame  that  will  decrease  cost  in  the  future.  For  example,  investment  in  a  test  technology 
that  decreases  the  test  cost  while  increasing  risk  reduction  is  a  cost  avoidance  action.  A 
productivity  improvement  is  a  reduction  in  future  personnel  time  and  effort  requirements 
associated  with  a  function  or  assigned  task  that  has  been  included  in  an  approved 
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program.  Under  normal  circximstances,  productivity  improvements  do  not  represent  an 
opportunity  to  reduce  an  approved  program,  budget  or  force  stmcture.  Unlike  cost 
avoidance,  productivity  improvements  have  no  direct  impact  on  future  requirements  for 
funding,  but  enable  accomplishment  of  more  work  with  existing  personnel.  Productivity 
improvements  can  accrue  at  any  time  during  the  life  cycle. 

b.  Cost  Estimation  Approaches 

With  a  description  of  the  process,  contents  and  metrics  for  cost  estimation, 
the  analyst  must  select  an  approach.  “The  engineering  approach,  parametric  approach, 
analogy  approach,  and  expert  opinion  approach  are  four  cost-estimating  methods.  The 
use  of  a  specific  approach  varies  with  the  reliability  and  quantity  of  available  data.  Each 
approach  has  limitations.”  (CEAC,  1997) 

The  engineering  (bottom-up)  approach  is  an  examination  of  separate  work 
segments  in  detail  and  a  synthesis  of  the  many  detailed  estimates  into  a  total.  With  this 
approach,  the  analyst  divides  the  system,  activity,  or  item  of  hardware  into  its  segments 
and  makes  an  estimate  of  each  segment's  costs.  The  analyst  then  combines  these 
estimated  costs  with  estimates  of  integration  costs  to  arrive  at  a  total  cost.  A  major 
limitation  of  the  engineering  approach  is  that  it  requires  the  analyst  to  have  an  extensive 
knowledge  of  the  system,  activity,  or  item.  Also,  the  analyst  must  know  both  the 
development  and  production  processes.  Particularly  for  new  technologies,  the  detailed 
knowledge  required  for  a  complete  engineering  analysis  is  not  always  available,  making 
this  approach  the  most  difficult  to  apply.  (CEAC,  1997)  Since  the  technologies  are 
known  for  the  EPLRS,  AC  AD  A,  and  Abrams  M1A2  historical  studies  the  engineering 
approach  was  used  in  the  analysis. 
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The  benefit  estimating  process  is  similar  to  that  for  cost  estimating.  Data 
must  be  collected  from  appropriate  sources  and  analyzed;  relationships  among  data  must 
be  identified.  Inflation  and  discounting  must  be  applied  to  annual  dollar  values  via 
standard  methods;  the  economic  life  of  the  alternatives  and  the  fiscal  years  when  benefits 
accrue  must  be  carefully  considered.  Once  benefits  have  been  calculated,  savings  must 
be  separated  from  cost  avoidance  and  productivity  improvements. 

c.  VPG  Considerations 

In  the  VPG  cost  estimating  methodology,  DTC  wishes  to  separate 
operating  costs  from  fixed  or  investment  costs.  The  total  cost  of  the  current  alternative  = 
Fixed  Costs(  Fa)  +  Variable  Costs  (Va).  DTC  wishes  to  make  cost  estimates  for  the  PMs. 
The  fixed  cost  to  the  PM  may  be  zero  if  an  acquisition  program  can  use  a  maintained 
facility  developed  by  a  predecessor  program,  or  new  capability  is  developed  by  DTC. 
For  virtual  testing,  the  fixed  cost  is  usually  an  investment  in  the  model,  simulator  or 
stimulator.  The  Investment  Cost,  a  nonrecurring  cost,  includes  land,  building,  machinery 
and  equipment.  Operating  Cost,  or  recurring  cost  includes  personnel,  maintenance,  and 
materiel/supplies.  Fa  includes  instrumentation  development  cost,  and  facility 
development  cost  (materials  and  labor).  Va  includes  labor,  expendables,  and  maintenance 
and  test  preparation/planning. 

Cost  estimates  for  the  VPG  are  complicated  due  to  the  cost  estimation  of 
software  development.  “Because  software  life  cycle  costs  account  for  a  significant 
portion  of  information  systems'  costs,  and  are  often  significant  in  materiel  systems,  they 
must  be  estimated  carefully.  Software  cost  estimating  involves  a  large  degree  of 
professional  judgment,  from  both  a  project  management  and  cost  analysis  perspective.” 
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(CEAC,  1997)  To  the  extent  that  each  program  requires  a  degree  of  VPG  ongoing 
software  and  technology  development,  the  cost  estimating  process  requires  care. 

Since  one  of  the  motivations  behind  doing  virtual  testing  is  accomplishing 
testing  faster  and  cheaper,  a  cost  avoidance  methodology  would  help  define  the  benefits 
associated  with  this  new  testing  paradigm.  Based  on  the  previous  sections  that  help 
define  what  physical  and  virtual  testing  involve,  a  cost  avoidance  methodology  can  be 
developed.  In  order  to  develop  a  VPG  cost  avoidance  methodology,  the  basic  steps  for 
economic  analysis  must  be  considered.  The  Seven  Steps  for  Economic  Analysis  include: 
establish  and  define  goals  and  objectives,  formulate  assumptions  and  identify  constraints, 
identify  alternatives  for  meeting  objectives,  determine  inputs  and  outputs  of  each 
alternative,  compare  cost  and  benefits  of  alternatives,  evaluate  and  determine  the  risk  and 
uncertainties,  prepare  conclusion  and  recommendation.  For  this  effort  the  focus  will  be 
to  compare  the  cost  and  benefits  of  alternatives. 

When  performing  a  benefit  analysis,  all  significant  benefits  must  be 
included  whether  quantifiable  or  non-quantifiable.  Benefits,  which  cannot  be  quantified, 
should  be  described  in  narrative  form.  Every  effort  should  be  made  to  quantify  benefits 
to  the  maximum  extent  possible.  A  sample  of  some  benefits  to  PMs,  testers  and 
evaluators  using  M&S  include:  performing  the  mission  with  fewer  test  items,  increased 
understanding  of  simulation  technology,  supporting  safety  and  environmental 
assessments,  preventing  false  starts  by  performing  test  rehearsals,  better  understanding  of 
how  the  components/subsystems/systems  under  test  are  designed  to  function  in  an 
integrated  fashion,  more  confidence  in  data  due  to  verification  by  simulation,  and 
providing  a  test  diagnostic  capability.  Some  of  the  benefits  of  using  M&S  to  the 
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customer  or  Program  Manager  include:  fewer  test  items,  lower  test  costs,  reduction  in  test 
time,  early  knowledge  of  test  environment  models,  powerful  tool  for  data  analysis  and 
trouble  shooting,  improved  data  requirements  generation,  and  validation  of 
component/system  models.  Once  the  benefits  have  been  identified,  then  the  benefits  are 
subdivided  into  those  that  are  dollar  quantifiable  and  those  that  are  quantifiable  in  other 
terms. 

The  benefit  estimating  process  is  similar  to  that  for  cost  estimating.  Data 
must  be  collected  from  appropriate  sources  and  analyzed;  relationships  among  data  must 
be  identified;  inflation  and  discounting  must  be  applied  to  annual  dollar  values  via 
standard  methods;  the  economic  life  of  the  alternatives  and  the  fiscal  years  when  benefits 
accrue  must  be  carefully  considered.  Once  benefits  have  been  calculated,  savings  must 
be  separated  fi'om  cost  avoidances  and  productivity  improvements. 

The  three  categories  of  quantifiable  benefits  include  cost  savings,  cost 
avoidance  and  productivity  improvements.  A  cost  savings  results  in  the  reduction  of  an 
approved  program.  The  item  under  consideration  must  be  part  of  an  approved  budget.  A 
cost  avoidance  is  the  result  of  an  action  taken  in  the  immediate  time  fi-ame  that  will 
decrease  cost  in  the  future.  For  example,  an  engineering  improvement  that  increases  the 
mean  time  between  failures  and  thereby  decreases  operation  and  maintenance  costs  is  a 
cost  avoidance  action.  A  productivity  improvement  is  a  reduction  in  future  personnel 
time  and  effort  requirements  associated  with  a  function  or  assigned  task  that  has  been 
included  in  an  approved  program.  Under  normal  circumstances,  productivity 
improvements  do  not  represent  an  opportunity  to  reduce  an  approved  program,  budget  or 
force  structure.  Unlike  cost  avoidances,  productivity  improvements  have  no  direct 
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impact  on  future  requirements  for  funding,  but  enable  accomplishment  of  more  work 
with  existing  personnel.  Productivity  improvements  can  accrue  at  any  time  during  the 
life  cycle.  Following  is  a  cost  avoidance  methodology  developed  to  support  the  cost 
avoidance  activities  of  the  VPG  for  test  articles  as  well  as  test  resources. 

The  developed  VPG  cost  avoidance  methodology  uses  a  comparison 
analysis  between  the  cost  of  the  current  alternative  and  the  cost  of  the  simulation 
alternative.  This  is  similar  to  the  business  savings  calculation  in  Figure  1  (Benefit  Cost 
Calculations). 


Total  Cost  of  Current  Alternative  (physical  testing)  =  Fixed  Costs(  Fa)  + 
Variable  Costs(Va);  where  the  fixed  cost  may  be  zero.  Total  Cost  for  Simulation 
Alternative  (virtual  testing)  =  Fixed  Costs  (Fb)+  Variable  Costs  (Vb);  where  the  fixed  cost 
is  usually  an  investment.  Cost  Avoidance  =  Total  Cost  of  Current  Alternative  -  Total 
Cost  for  Simulation  Alternative. 

Where: 

•  Investment  Cost  (Nonrecurring  cost)  -  Land,  building,  machinery, 
equipment  and  software. 

•  Operating  Cost  (recurring  cost)  -  Personnel,  maintenance/updates 
(hardware  and  software),  and  materiel/supplies. 

•  Fa  -  Instrumentation  Development  Cost,  Facility  development  cost 
(materials  and  labor). 

•  Va  —  Labor,  expendables,  maintenance  and  test  preparation/planning. 

•  Fb  -  Simulation  development/modification,  instrumentation  development, 
HWIL  simulation  facility  development. 

•  Vb  -  Labor  for  test  execution,  expendables,  hardware  and 
software/simulation  maintenance,  test  preparation/planning. 

If  the  overall  objective  is  to  field  faster,  better,  and  cheaper  weapon 


systems,  then  a  true  measurable  metric  would  be  any  numerical  value  that  enables  us  to 
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assess  how  much  faster,  how  much  better,  and  how  much  cheaper  a  given  acquisition 
process  is  or  can  be.  Some  quantifiable  measurable  metrics  are  noted  herein. 

•  Cost  with  consideration  for  the  consumable  item  price  index  and  military 
specification  conversion  price  benefit. 

•  Schedule  with  consideration  for  the  acquisition  phase  time,  administrative 
lead  time  and  test  time. 

•  Program  cost  with  consideration  for  change  in  program  cost  as  a 
consequence  of  changed  acquisition  processes. 

•  Unit  Life-Cycle  Cost  considering  change  in  projected  unit  life-cycle  cost 
as  a  consequence  of  changed  acquisition  process. 

•  Operational  performance  versus  cost  with  consideration  for  the 
comparison  of  operational  test  results  versus  specified  performance. 

•  Cost  of  Performance  considering  the  kind  of  system  performance  that  can 
be  bought  for  a  given  cost.  To  derive  this  metric  it  would  be  necessary  in 
some  way  to  quantify  various  combinations  of  system  performance. 

•  Cost  as  an  independent  variable  is  savings  in  a  program  where  costs  are 
held  constant  and  performance  and  schedule  are  adjusted. 

The  primary  metrics  used  in  the  following  studies  considered  were  cost 

and  schedule  even  though  others  could  be  considered  for  future  analysis. 

4.  Step  4.  Evaluate  Alternatives 

a.  Analysis  of  Historical  Programs 

Where  possible,  the  live  test  equivalent  to  a  virtual  test  procedure  was 
chosen  or  designed  to  provide  the  same  level  of  risk  reduction.  It  is  not  assumed  that  test 
money  is  limitless.  Thus,  if  the  live  test  design  produced  an  unreasonable  test  cost,  a 
reduced-scope  live  test  was  used  for  comparison,  with  an  attendant  increased  technical 
risk.  Labor  costs  are  normalized  to  the  same  year,  and  not  broken  down  by  skill  levels. 
There  is  a  factor  for  inflation.  The  cost  of  facilities  may  be  paid  with  DTC  institutional 
funding. 
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b. 


Enhanced  Position  Location  Reporting  System  Study 


The  EPLRS  Technical  Test  III  (TT  III)  was  conducted  at  the  US  Army 
Electronic  Proving  Ground  (EPG),  Fort  Huachuca,  AZ.  A  series  of  EPLRS  tests  (TT  I 
and  TT  II)  were  conducted  five  years  prior  to  TT  III,  using  live  testing  (i.e.,  human  radio 
operators/data  collectors).  The  methodology  used  was  to  calculate  the  additional 
resources  required,  for  the  same  number  of  test  days  (75)  during  TT  III,  using  TT  II 
(labor  intensive)  methods. 

The  TT  I/II  test  consisted  of  an  average  of  160  radio  sets  (RSs)  distributed 
at  various  locations  in  an  area  of  400  km  and  required  over  300  operators/data  collectors. 
There  was  limited  automated  data  collection.  Test-scenario-control  was  by  hand-held 
FM  radio,  and  required  extensive  logistics  support  for  the  operators.  All  of  the 
equipment  at  each  site  was  installed  and  removed  daily  for  security  purposes,  which 
lengthened  the  test  day  by  about  four  hours.  Data  consolidation,  reduction,  and  analysis 
normally  took  about  five  days. 

During  TT  III  many  time/cost-saving  automated  tools  and  methods  for 
control,  monitoring,  and  data  collection  were  used.  Elements  contributing  to  cost 
avoidance  were:  less  logistics  support;  number  of  operators;  a  Test  Control  Center 
(TCC),  use  of  a  Personal  Computer  (PC)  as  a  Test  Item  Stimulator  (TIS);  and 
security/test  sheds  with  alarms  to  house  the  equipment.  The  TCC,  with  automated  data 
collection,  reduced  analysis  time  from  five  days  to  one.  Each  of  the  average  of  120  sheds 
held  a  PC/TIS,  the  EPLRS,  a  packet  radio  for  commimications  between  the  TIS  and  the 
TCC,  and  the  alarm.  In  effect,  the  self-contained  sheds  with  the  TIS  were  a  "virtual 
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workforce"  that  replaced  the  human  operators/data  collectors.  Shed  support  consisted  of 
exchanging  batteries,  turning  on/off  equipment,  TIS  data  dump,  and  on-call  maintenance. 


The  original  study  (Brugh,  1997  A)  assumed  that  the  TCC,  sheds,  and 
automated  data  reduction  tools  were  not  available.  Daily  setup  and  tear  down  increased 
the  workday  by  two  hours  and  required  large  trailers  for  equipment  storage.  Support 
personnel  increased  by  50%  from  69,  plus  120  operators.  Transport  vehicles  carried  six 
testers  and  test  equipment.  One  instructor  was  required  per  group  of  1 6  to  provide  three 
days  of  training  to  military  and  civilian  operators.  The  analysis  constrained  the  test  cost 
at  $10M  maximum,  and  assumed  a  civilian/military  mix  of  testers  to  be  115/40.  Civilian 
labor  was  $32/hr  and  military  labor  was  free  to  the  customer.  The  analysis  did  not 
consider  the  test  center’s  investment  in  PCs,  packet  radios,  automated  data  collection 
software,  and  the  cost  of  the  TCC. 

The  additional  resource  cost  for  75  test  days  was  $2.2M  labor  +  $0.8M 
overtime  +  $0.1M  equipment,  and  fixed  costs  of  $0.1  M  training  +  $.03  M  equipment  for 
a  total  $3.2M.  Since  the  TT  III  cost  $6.3M  as  conducted,  the  $10M  spending  limit  would 
be  nearly  reached  ($3.2M  -i-  $6.3M)  during  the  75  test  days  of  labor  intensive  testing. 
Expert  opinion  was  (Brugh,  1 997)  that  only  one-third  to  one-half  the  amoimt  of  usable, 
high-quality  data  would  be  acquired  during  the  75  test  days  with  this  method.  Another 
alternative  to  the  earlier  study  allows  use  of  the  sheds,  which  would  avoid  the  overtime 
costs.  This  still  results  in  a  $2.4M  cost  avoidance  for  the  VPG  stimulator  technology. 

Using  the  VPG  cost  avoidance  methodology  presented  previously,  the 

following  summaries  can  be  made.  The  cost  of  the  EPLRS  TT  I/II  is  $10.085M.  The 

cost  of  the  EPLRS  TT  III  is  $7.233M.  Both  test  cost  amounts  have  been  adjusted  to  1993 

80 


base  year  dollars.  These  figures  were  obtained  by  totaling  the  fixed  costs  of  facilities  and 
instrumentation  and  variable  costs  of  vehicles,  hand  held  radios  and  labor.  Subtracting 
the  cost  of  the  virtual  test  from  the  physical  test,  the  cost  avoidance  is  $2.852M.  The 
following  savings,  which  used  an  estimated  test  costing  analysis,  were  stated  in  the  study: 
“EPLRS  Savings  as  tested:  Virtual  testing  during  TT  III,  saved  the  EPLRS  program  $3.2 
million,  provided  many  millions  of  dollars  worth  of  risk  mitigating  and  risk  management 
information,  and  shortened  testing  by  a  minimum  of  75  days.”  (Brugh,  1996)  Using  the 
VPG  cost  avoidance  methodology  produces  numbers  that  are  marginally  close  to  those 
stated  in  the  study.  See  the  test  methodology  comparison  in  Table  3. 


Human  in  the  Loop 

Test  Item  Stimulator 

Required  Personnel 

155 

69 

Test  Days  Avoided 

0 

! 

75 

Cost  To  Test  $ 

$10.085M 

$7.233M 

Cost  Avoidance  $2.852M 

Table  3. 

EPLRS  Test  Methodoi 

logy  Comparison.  (Soince:  Researcher) 

In  summary,  virtual  testing  during  TT  III  resulted  in  substantial  cost 
avoidance,  $2.4M  -  $3.2M,  and  significantly  increased  quality  and  quantity  of  data,  vice 
use  of  less  VPG-intensive  technologies. 

c.  Advanced  Chemical  Agent  Detector  Alarm  Study 
The  chemical/biological  defense  case  study  was  on  the  ACADA  PVT  at 
Dugway  Proving  Ground,  UT.  As  might  be  expected,  open  air,  live  field-testing  with 
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chemical  agents  is  neither  practical  nor  feasible.  Chemical  agent  detector  testing  in  the 
past  has  essentially  always  been  conducted  in  environmentally  controlled  chambers,  or 
using  simulants  in  lieu  of  live  agents  (i.e.,  virtual  testing).  Previous  testing  was 
conducted  in  Building  3445,  the  large  chamber  facility.  While  the  large  chamber  can 
accommodate  testing  of  large  vehicles  and  equipment,  testing  of  small  agent  detectors 
required  construction  of  a  smaller  chamber  inside  the  large  facility.  Past  testing  indicated 
that  an  earlier  small  chamber  design  could  simultaneously  test  only  two  detectors  of  the 
six  for  which  it  was  designed.  The  number  of  support  personnel  for  Building  3445  is 
fourteen.  For  the  ACADA  PVT,  a  new  stand-alone,  HWIL  test  facility  (the  Detector  Test 
System,  DTS)  capable  of  testing  twelve  detectors  simultaneously,  and  requiring  only  five 
testers,  was  developed.  The  customer  contributed  to  the  fixed-cost  of  construction  of  the 
DTS  ($82,200  of  a  total  cost  of  $421,000).  Twelve  detectors  were  tested  simultaneously 
for  39  days  in  Phase  I,  eight  for  92  days  during  Phase  II. 

The  study  objective  (Brugh,  1997)  was  to  estimate  cost  for  Phase  I  Agent 
Testing  and  Phase  II  Agent/Simulant  testing  during  the  ACADA  PVT,  using  the  DTS  and 
using  Building  3445.  Testing  using  a  corrected  six  detector  chamber  design  was  not 
considered. 

The  study  assumptions  (Brugh,  1997)  were  as  follows.  Each  tester  costs 
$355  per  ten-hour  day.  The  times  required  to  startup,  conduct  and  shut  down  a  trial  are 
the  same  in  both  facilities.  The  PM  had  no  fixed  facility-usage  cost  and  no  cost  for 
construction  of  a  small  chamber  in  Building  3445.  Test  equipment  reliabilities  are  the 
same  in  both  cases. 
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The  operating  cost  is  the  number  of  people  who  run  the  facility  multiplied 
by  the  personnel  cost  per  day,  multiplied  by  the  number  of  test  days.  For  phase  I,  the  cost 
is  $.065M.  The  corresponding  Building  3445  cost  would  be  greater  by  factors  of  14/5 
(personnel)  and  12/2  (detectors)  to  total  $1.1M.  For  Phase  II,  the  92-day  cost  was 
$.154M.  The  corresponding  Building  3445  cost  would  be  greater  by  factors  of  14/5 
(personnel)  and  8/2  (detectors)  or  $1.7M. 

The  variable  cost  totals  are  (Cost  Phase  I  +  Cost  Phase  II),  $.219M  for  the 
DTS  and  $2.8M  for  Building  3445.  The  total  cost  to  the  PM,  fixed  plus  variable,  was 
$.302M  for  the  DTS.  The  test  schedule  would  increase  from  131  days  for  the  DTS  to,  in 
Building  3445,  602  days.  For  this  study,  we  add  an  alternative  scenario  that  assumes 
correction  of  the  earlier  design  to  allow  simultaneous  measurement  of  six  detectors  in 
Building  3445.  This  results  in  $.364M  variable  Phase  I  costs  and  $.575M  in  Phase  II,  or 
$.939M  total  cost.  Schedule  would  be  201  test  days.  Investment  in  the  DTS  resulted  in 
substantial  cost  avoidance  to  the  customer  even  for  this  technically  unsubstantiated 
scenario. 

The  objective  of  the  AC  AD  A  study  was  to  compare  the  cost  and  schedule 
of  the  virtual  testing  during  the  ACADA  Production  Verification  Testing,  with  the  same 
amount  of  testing  using  previous  testing  techniques.  As  one  would  expect,  live  open  air 
testing  with  chemical  agents  is  not  practical  or  feasible.  Chemical  agent  detector  testing 
in  the  past  has  essentially  always  been  conducted  in  environmentally  controlled 
chambers.  This  technique  is  essentially  virtual  testing.  The  comparison  was  made 
between  the  cost  and  schedule  during  the  ACADA  production  verification  testing  using 
the  DTS  and  the  cost  and  schedule  if  the  same  tests  had  been  conducted  using  the 
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Chemical  Agent  Monitor  test  configuration.  The  DTS,  which  uses  more  automated 
simulation  techniques,  will  be  considered  the  virtual  testing  method.  The  testing 
completed  using  Chemical  Agent  Monitor  test  configuration  will  be  considered  the 
physical  testing  technique.  The  factors  considered  when  conducting  a  cost  and  schedule 
analysis  of  the  two  testing  techniques  include:  number  of  persoimel  required  to  conduct 
testing,  number  of  items  which  can  be  tested  simultaneously,  time  required  for  test  setup, 
and  facility  costs  (procurement,  setup,  and  sustaining). 

Again,  using  the  cost  avoidance  methodology  presented  previously,  the 
following  summaries  can  be  made.  The  cost  of  the  ACADA  test  using  the  Detector  Test 
System  is  $.302M.  This  figure  was  calculated  using  the  fixed  cost  of  developing  the 
Detector  Test  System  and  the  variable  cost  of  labor.  The  cost  of  the  ACADA  test,  if 
conducted  using  the  Chemical  Agent  Monitor  test  configuration,  would  be  $2.823M. 
This  amoimt  was  calculated  using  the  variable  cost  of  labor  for  running  a  test  using  the 
Chemical  Agent  Monitor  test  configuration.  Subtracting  the  cost  of  the  virtual  test  from 
the  physical  test,  the  cost  avoidance  is  $2.521M.  The  folloAving  savings,  which  used  an 
estimated  test  costing  analysis,  were  stated  in  the  study:  “ACADA  Savings  as  Tested: 
Virtual  testing  during  Production  Verification  Testing,  saved  the  ACADA  program  $2.5 
Million  and  shortened  testing  by  471  days.”  (Brugh,  1997).  The  numbers  for  cost  savings 
stated  in  the  ACADA  study  matched  the  numbers  for  the  calculations  using  the  VPG  cost 
avoidance  methodology.  The  $2.5  Million  cost  savings  stated  in  the  study  are  actually 
cost  avoidance  dollars  by  definition.  See  the  test  methodology  comparison  in  Table  4. 


84 


Chemical  Agent  Monitor  Test 

Detector  Test  Set 

Detectors  Tested 

2 

12 

Test  Days 

602 

131 

Cost  To  Test  $ 

$2.823M 

$.302M 

Cost  Avoidance  $2.521M 

Table  4.  ACADA  Test  Methodology  Comparison.  (Source:  J 

Researcher) 

The  results  were  substantial  cost  avoidance,  $2.6M,  and  schedule 
avoidance,  471  test  days,  from  investment  in  the  newer  simulation  technology  vice  usage 
the  existing  old  technology. 

d.  Abrams  M1A2  Tank  Study 

This  study  (Brugh,  1996)  identified  use  of  virtual  testing  during  the 
Production  Qualification  Test  (PQT)  progr^  and  during  the  live  fire  testing  at  Aberdeen 
Test  Center  (ATC).  Seven  PQT  tests  at  ATC  and  two  at  the  White  Sands  Missile  Range 
(WSMR)  used  virtual  testing  that  resulted  in  cost  avoidance.  The  WSMR 
Electromagnetic  Environmental  Effects  test  and  Nuclear  Effects  test  had  no  feasible  live 
test  alternative  and  therefore  did  not  involve  cost  avoidance,  but  did  result  in  risk 
reduction.  For  this  work,  only  the  accuracy  test  will  be  addressed. 

Within  the  accuracy  test  description,  the  first  letter  indicates  tank  motion 
and  the  second  indicates  target  motion  to  be  stationary  (S)  or  moving  (M).  S/M,  M/M, 
and  M/S  at  Aberdeen  Test  Center’s  Trench  Warfare  I  (TW  I),  with  targets  simulated  by 
laser,  each  costs  15  testers  @  $400/tester/day  (10  hr)  for  22  days,  or  $.132M.  Non-Firing 
Tracking  M/M  (NFT  M/M)  costs  10  testers  @  $400/tester/day  for  8  days  or  $.032M.  The 
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total  labor  cost  on  TW  I  is  $.428M.  Expert  opinion  indicates  that  physical  target 
simulators  require  at  least  twice  as  many  days  to  collect  the  required  data  and  two 
additional  testers.  A  daily  target  cost  of  $500/target/day  would  accrue  for  the  S/M,  M/M, 
M/M  NET  scenarios.  For  the  S/M  and  M/M  tests,  the  cost  would  be  $.32 IM.  The  M/S 
cost  is  the  same  ($.132M).  The  M/M  NFT  cost  would  be  $.085M.  The  total  cost,  labor 
and  targets  of  the  physical  simulation  tests  are  $.859M.  The  difference  (S/M,  M/M,  M/S 
and  M/M  NFT)  of  labor  and  targets  is  $.43 IM.  See  the  test  methodology  comparison  in 
Table  5. 


Physical  Target 

Laser  Target 

Projectiles  Saved 

0 

2580 

Test  Days  Avoided 

0 

258 

Cost  To  Test  $ 

$.859M 

$.428M 

Cost  Avoidance  $4.8M 

Table  5.  Abrams  Ml  A2  Test  Methodology  Comparison.  (Source:  Researcher) 


For  S/S  at  Trench  Warfare  II,  there  was  a  requirement  for  measuring 
accuracy  at  four  distances.  Prior  to  TW  II  construction,  firing  projectiles  at  each  of  the 
required  distances  met  the  requirement.  At  TW  II,  virtual  video  “scoring”  at  the  four 
distances,  for  a  -projectile  fired  at  the  longest  distance,  reduced  by  75%  the  number  of 
projectiles  and  the  test  time.  This  avoided  almost  $3M  in  the  cost  of  ammunition  alone, 
with  added  labor  reduction  of  nearly  $1M.  The  total  cost  avoidance  by  using  virtual 
testing  during  the  accuracy  testing  was  $4.4M.  Other  virtual  fire  control  tests  brought  the 
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total  cost  avoidance  to  $4.8M,  a  schedule  reduction  of  258  days,  and  2580  projectiles  not 
fired. 

e.  Simulation/T est  Acceptance  Facility  (STAF) 

The  STAF  is  a  HWIL  facility  at  the  Redstone  Technical  Test  Center 
(RTTC)  in  Huntsville,  AL  that  greatly  reduces  the  number  of  destructive  live  flights  of 
missiles  in  production  flight-testing  and  in  stockpile  testing.  Completed  missiles 
containing  tactical  seekers,  guidance  electronics,  inertial  navigation  systems,  warheads, 
squibs,  motor,  and  control  actuators  are  tested  in  a  remotely  controlled  bunker.  With  the 
motor  and  warhead  circuits  disconnected,  the  missile  flight  dynamics  are  simulated  using 
a  six  degree  of  freedom  digital  model  of  the  missile’s  airfirame  running  in  real  time.  The 
facility  can  modulate  radio  frequency  signals  to  present  realistic  in-band  representations 
of  complex  targets  to  the  millimeter  wave  seeker  or  realistic  InffaRed  (IR)  signals  to  an 
IR  seeker.  A  real-time  data  collection  system  records  data  from  simulated  launch  to 
simulated  target  impact.  Missiles  are  consumed  only  to  test  motors  and  warheads,  and  as 
model  validation.  Rounds  tested  in  the  facility  can  be  returned  to  the  inventory.  Separate 
cost  estimates  (Johnson,  1996),  not  repeated  for  this  work,  indicate  that  the  STAF  with 
RF  representation  cost  $6M.  Typical  live  firing  lot  acceptance  testing  costs  customers 
approximately  $9.5M/year.  The  same  testing  conducted  in  the  STAF  costs 
approximately  $1 .5M/year  for  cost  savings  of  $8M/year. 
f.  VPG  Cost  Avoidance 

The  chart  shown  below  summarizes  the  VPG  cost  avoidance  to  the  customer 
through  use  of  M&S  to  support  the  four  acquisition  programs  addressed  in  this  thesis.  , 
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COST  AVCffDANCE  (SIV^ 


Figure  4.  VPG  Cost  Avoidance.  (Source:  Researcher) 


5.  Step  5.  Conduct  Sensitivity  Analysis 

The  STAF  facility  was  the  only  program  that  performed  a  detailed  sensitivity 
analysis.  The  Air  to  Ground  Missile  System  (AGMS)  Project  Office  and  DTC  RTTC 
performed  an  analysis  of  the  perspective  factors  and  trade-offs  involved  in  implementing 
a  HWIL  lot  acceptance  program  versus  a  traditional  Fly-to-Buy  (FTB)  program.  This 
analysis  considered  the  tw^o  alternatives  available  to  Longbow  Hellfire  (LBHF),  as  well 
as,  the  historical  data  provided  by  the  earlier  HELLFIRE  I.  The  primary  factors  in  the 
development  of  the  assumptions  involved  in  this  analysis  process  were  as  follows: 

•  Unit  Costs: 

•  HELLFIRE  cost  per  missile  S  20k. 

•  LBHF  cost  per  missile  $  300k  @  Low  Rate  Initial 

Production  (LRIP). 

•  Test  Programs: 
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•  HELLFIRE  and  LBHF  Lot  Acceptance  Testing:  FTB  @4-10 
missiles  per  month. 

•  LBHF  HWIL  Simulation:  4  live  fires  per  year  and  up  to  20 
missiles  simulated  flight  testing  per  month. 

•  Facilities:  Facility  for  the  HWIL  Simulation:  $  5.8  million.  The  building 
and  instrumentation. 

•  Labor:  Personnel  needed  to  run  this  facility  for  one  year:  estimated  at 
approximately  $0.8  million. 

With  these  factors  in  mind,  a  trade-off  analysis  was  performed  using  costs  for  four 
missiles  with  the  results  portrayed  in  Table  2.  The  total  savings  projected  under  a  HWIL 
simulation  lot  acceptance  test  program  would  represent  cost  savings  of  $7.78  million  per 
year  over  a  conventional  FTB  program.  Even  including  the  $5.8  million  necessary  for 
developing  the  HWIL  simulation  facility,  the  payback  period  under  this  assumption 
would  be  less  than  one  year. 


LONGBOW 

Lot  Acceptance  Methodology 

Cost  Tradeoff 

(Post  Cost  Reduction  Program) 

BASIC  HE 

LBHF 

SIMULATED 

FTB 

FTB 

PROGRAM 

(What  if) 

MISSILE  COST 

$1.15  MIL 

$8.11  MIL 

$0.68  MIL 

EGLIN  SUPPORT 

$0.72  MIL 

$0.72  MIL 

$0.06  MIL 

RTTC  SUPPORT 

$0.53  MIL 

$0.53  MIL 

$0.44  MIL 

SYS.  SIM.  SUPPORT 

$0 

$0 

$0.40  MIL 

TOTAL 

$2.40  MIL 

$9  MIL 

$1.58  MIL 

SAVINGS  =  $9.36  MIL 

NONRECURRING  FACILITY  COSTS: 

$1.58  MIL 

Total  Facility  Cost: 

PM-ITTS  $1,000K 

$7.78  MIL  PER  YEAR 

other  S330K 

AGMS-PMO  $4,500K 

PAYBACK  PERIOD  «  1  YEAR 

S5,830K 

Table  2.  Longbow  Lot  Acceptance  Methodology  Cost  Tradeoff.  (Johnson,  1996) 
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The  Basic  Hellfire  (HF)  FTB  coliimn  indicates  approximately  what  AGMS  pays 
per  year  for  the  HELLFIRE  I  FTB  program,  and  also  indicates  where  the  true  cost  growth 
in  FTB  actually  lies,  namely,  missile  cost.  This  total  amount  includes  missile  cost,  range 
support,  and  pre-flight  support.  The  LBHF  FTB  column  indicates  what  the  cost  would  be 
if  the  AGMS  Office  implemented  a  similar  FTB  program  with  just  four  missiles  per 
month.  The  tremendous  yearly  cost  avoidance  is  due  to  the  non-destructive  nature  of  the 
simulated  flights,  which  allows  the  rounds  to  be  placed  in  inventory  upon  test 
completion.  The  analysis  indicates  a  facility  cost  payback  period  of  much  less  than  one 
year. 

To  determine  an  annual  return  on  investment  for  the  STAF  facility,  the  ROI 
formula  in  Chapter  3  of  this  thesis  was  used.  ROI  =  cash  inflow  (Benefit)  /  net 
investment.  To  account  for  the  test  and  simulation  environment  the  ROI  formula  was 
rephrased  as  follows:  ROI  =  test/simulation  workload  /  fixed  cost  of  facility  and 
instrumentation.  So,  for  the  STAF  facility  the  ROI  =  (4  live  fires  &  20  missiles/month 
for  12  months)  /  facility  and  instrumentation  costs  =  $1.58  M  /  $5.8  M  =  .2724.  This 
represents  the  annual  ROI  for  the  STAF  facility. 

6.  Step  6.  Develop  a  Migration  Strategy 

Of  the  four  VPG  projects  examined  in  this  thesis,  only  the  use  of  the  STAF 
facility  was  fully  integrated  into  its  associated  acquisition  program.  The  STAF  facility  is 
an  integral  part  of  the  lot  acceptance  program  for  the  Longbow  missile.  Unfortunately, 
formalized  management  goals  and  a  systematic  approach  to  capturing  the  expected 
benefits  of  the  progreim  were  not  developed.  Based  on  this  lack  of  formalized  goals,  the 
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monitoring  and  feedback  process  is  limited  to  the  future  possibilities  for  use  of  this 
technology  for  program  savings  in  general. 

7.  Step  7.  Monitoring  the  Process  and  Assessing  Results  through 
Formalized  Feedback 

Other  Army  programs  have  been  approached  about  using  the  STAF  facility  on  a 
time-sharing  basis  and  using  HWIL  simulation  to  reduce  testing  costs.  The  key  factor 
here  is  the  need  to  highlight  the  necessity  of  early  planning  to  accomplish  this  task. 
LBHF  has  already  demonstrated  that  it  is  much  easier  to  design  features  into  the  missile 
that  lend  themselves  to  aiding  the  ease  of  HWIL  testing,  than  attempting  to  correct  these 
after  the  missile  design  has  been  locked  in.  In  the  case  of  LBHF,  the  move  to  HWIL 
simulation  meant  determining  an  after  the  fact  means  to  attach  the  simulation  hook  up 
cable  through  a  sealed  access  panel  in  the  missile  body.  In  addition  numerous  signal  and 
interface  problems  and  modifications  were  needed.  These  requirements  would  have  been 
easier  to  resolve  if  they  had  been  included  in  the  original  system  design  to  accommodate 
a  HWIL  approach.  Future  systems  should  investigate  this  area  dming  the  design  phase, 
so  those  HWIL  critical  features  can  be  incorporated  into  the  design. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

The  management  of  acquisition  programs  is  one  of  the  most  difficult  jobs  in  the 
DoD  today.  When  making  complex  technological  decisions  where  there  are  a  limited 
number  of  variables  but  a- near  limitless  combination,  using  a  business  case  can  help  the 
decision-maker  see  through  the  fog.  M&S  allow  the  PM  to  experiment  with  a  larger 
number  of  possibilities  without  undue  risk  and  the  Ti&E  practitioner  more  opportunities 
to  provide  better  testing  at  a  lower  cost  and  schedule.  Once  the  essential  data  is 
determined,  collected,  and  then  shaped  within  a  model,  the  simulations  provide  a 
tremendous  advantage  over  traditional  methods  of  trial  and  error. 

The  use  of  M&S  is  no  longer  reserved  for  programs  on  the  cutting  edge  of 
technological  development,  but  rather  it  is  standard  practice.  Since  M&S  is  a  part  of 
everyday  practice,  both  industry  and  the  Government  should  rely  on  sound  business 
practices  to  ensure  the  successful  application  of  M&S.  Based  on  the  fact  that  M&S  is  an 
established  business  tool,  M&S  investment  justification  should  be  based  on  a  reasoned 
cost  benefit  analysis. 

Establishing  these  cost  benefit  relationships  can  however  be  just  as  difficult  as  the 
management  of  a  program  itself.  The  use  of  the  business  case  process  contained  in  this 
thesis  to  justify  M&S  development  provides  a  flexible  yet  structured  methodology  for 
decision-makers  to  weigh  alternatives.  This  business  case  analysis  permits  the  decision¬ 
maker  to  justify  investment  decisions  based  on  cash  flow  analyses,  as  a  function  of 

externally  imposed  constraints,  and  risk  reduction.  Additionally,  the  business  case  allows 
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the  capture  of  not  only  those  costs  and  benefits  that  are  internally  quantifiable  but  also 
those  that  are  non-quantifiable  with  respect  to  the  potential  benefits  that  may  exist 
external  to  the  program. 

When  the  business  case  was  applied  to  four  developmental  testing  historical 
cases,  some  poignant  discoveries  were  made.  When  a  structured,  disciplined  process  as 
outlined  in  this  thesis  was  applied  to  the  Enhanced  Position  Location  Report  System 
(EPLRS),  Advanced  Chemical  Agent  Detector  Alarm  (AC  AD  A),  ABRAMS  M1A2  and 
the  Simulation  Test  Acceptance  Facility  (STAF)  facility  study  data,  many  inadequacies 
were  observed.  Of  the  four  cases  analyzed,  the  STAF  facility  was  the  closest  to  using  a 
business-oriented  analysis. 

The  initial  baseline  step  of  the  business  case  process  proved  to  be  limited  in  the 
definition  of  metrics.  These  performance  measxires  for  the  four  cases  were  primarily 
limited  to  test  time  and  test  cost  (equipment  and  labor).  There  was  no  clear  statement  of 
assumptions  and  constraints.  The  baseline  did  however  address  areas  that  were 
impossible  or  difficult  to  test  physically.  This  initial  step  in  the  business  case  vividly 
displayed  that  there  truly  was  no  structured  planning  for  the  development  of  these  test 
and  simulation  technologies. 

The  vision  and  direction  step  of  the  business  case  process  is  marginally  well 
developed  by  the  Virtual  Proving  Groimd  Master  Plan.  This  plan  however  does  not 
establish  any  performance  criteria  or  measures  and  there  are  no  definite  goals  that  are  tied 
directly  to  any  fimding.  Additionally,  the  plan  is  very  high  level  in  nature  without  any 
specific  implementation  mechanism  or  projects.  The  plan  does  set  the  vision  for  using 
M&S  to  transform  developmental  testing. 
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The  steps  of  the  business  case  that  quantify  the  costs  and  benefits  of  alternatives 
and  perform  the  analysis  showed  that  a  cost  benefit  analysis  was  performed  in  order  to 
make  the  original  decision  to  develop  the  STAF  facility.  Unfortunately,  the  other  three 
cases  did  not  perform  a  documented  cost  benefit  analysis.  In  the  case  of  the  EPLRS, 
ACAD  A  and  the  ABRAMS  Ml  A2,  it  was  assumed  that  M&S  must  be  used  due  to  policy 
mandates  and  technology  advances.  The  use  of  M&S  in  the  EPLRS  testing  provided 
documented  personnel  multiplication  factors  that  resulted  in  a  test  time  reduction.  The 
ACADA  case  used  M&S  where  physical  testing  was  not  possible.  Additionally,  using 
M&S  to  test  the  ACADA  proved  to  increase  the  number  of  units  that  could  be  tested 
simultaneously  with  fewer  personnel.  The  ABRAMS  M1A2  testing  using  simulation 
provided  a  cost  avoidance  for  the  quantity  of  personnel  required,  a  reduction  in  test  time 
and  savings  from  not  firing  as  much  live  ammunition.  The  VPG  cost  avoidance 
estimated  for  these  programs  are  EPLRS  $2.4M,  ACADA  $2.6M,  ABRAMS  M1A2 
$4.8M  and  realized  for  the  STAF  $8M.  Other  tangible  benefits  include:  EPLRS 
shortened  testing  by  75  days,  ACADA  avoided  471  test  days,  ABRAMS  M1A2  realized 
a  schedule  reduction  of  258  days,  and  the  STAF  facility  realized  a  facility  payback  period 
of  one  year.  The  STAF  facility  enables  all  of  the  live  missiles  tested  to  be  put  back  in  the 
inventory  due  to  its  non-destructive  methods.  Additionally,  the  STAF  facility  is  able  to 
test  missiles  under  more  rigorous  and  controlled  conditions  with  fewer  resources  in  less 
time  than  typical  live  firing  programs. 

The  conduct  sensitivity  analysis  step  of  the  business  case  showed  that  the  STAF 
facility  was  the  only  case  that  used  any  type  of  sensitivity  analysis  to  determine  courses 
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of  action.  This  step  is  critical  to  comparing  the  options  and  making  a  good  business 
decision. 

The  migration  strategy  development  step  of  the  business  case  is  where  the 
implementation  plan  is  developed  to  enable  the  technology  or  project  to  be  integrated  into 
the  acquisition  program.  This  implementation  plan  needs  to  incorporate  a  systematic 
approach  to  ensme  the  developer  implements  the  identified  drivers  and  captures  the 
expected  benefits  to  be  assessed  at  the  post  implementation  review.  Of  the  four  projects 
analyzed,  the  STAF  facility  was  the  only  project  that  was  fully  integrated  into  its 
associated  acquisition  program  from  inception.  The  methodology  with  virtual  targets 
used  in  the  Abrams  Ml  A2  example  is  now  fully  integrated  into  all  fire  control  testing  for 
combat  vehicles  which  is  a  test  process  integration.  This  type  of  test  process  integration 
is  a  goal  of  the  Virtual  Proving  Groimd.  The  use  of  the  STAF  facility  to  perform  lot 
acceptance  testing  of  future  missile  lots  continues  to  be  integral  to  success  of  the 
Longbow  missile  program.  This  integration  was  planned  during  the  early  design  stages 
of  the  missile  program.  This  early  integration  enabled  and  ensured  that  the  STAF  facility 
would  successfully  support  the  Longbow  program.  Additionally,  the  benefits  of  the 
implementation  and  the  integration  of  the  facility  with  the  acquisition  program  are 
continually  being  tracked  for  future  development  of  a  STAF  facility  that  supports  other 
missile  sensor  technologies.  The  early  planning  and  migration  of  the  STAF  facility  has 
ensured  its  success  and  the  future  of  the  HWIL  technology  implementations  for  new 
programs. 

A  whole  generation  of  future  tactical  and  operational  level  missile  systems  could 
all  benefit  from  this  type  of  lot  acceptance  testing.  The  major  concern  here  is  that  these 
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systems  need  to  make  decisions  about  simulation  testing  now  while  still  in  their 
developmental  stages.  It  has  already  been  discovered  that  as  these  systems  increase  in 
complexity,  the  need  to  design  in  access  and  compatibility  with  simulation  testing  up 
front  is  crucial.  Delaying  this  activity  will  only  create  additional  design,  interface,  and 
cost  problems.  The  future  of  testing  will  use  simulation  assets  that  will  provide  benefit  to 
acquisition  programs.  This  benefit  can  only  be  realized  through  proper  early  planning  in 
the  design  of  both  development  and  testing  systems.  For  example,  this  could  be  achieved 
by  providing  high  fidelity  system  simulations  early  on  for  testing. 

The  monitoring  and  feedback  step  of  the  business  case,  Step  7,  is  intended  to 
monitor  and  assess  the  project’s  progress  towards  the  overall  vision.  The  STAF  facility 
is  the  only  project  that  has  monitored  their  process,  determined  ways  to  improve  the 
process  and  then  continued  towards  developing  future  facilities  based  on  their  concepts 
for  using  HWIL  technologies  to  improve  the  testing  capabilities  of  an  acquisition 
program.  By  making  an  investment  in  current  and  future  testing  technology  and  applying 
it  today,  avoidance  in  testing  costs  and  resources  can  be  realized  using  virtual  testing 
techniques. 

B.  RECOMMENDATIONS 

Having  a  clear  understanding  of  the  state  of  development,  the  following 
recommendations  will  serve  to  assist  decision-makers  with  the  development  of  M&S 
investment  strategies  based  on  a  sound  business  case  development  processes. 

PMs  must  be  encouraged  to  add  discipline  and  structure  to  their  M&S  justification 
process.  Service  leadership  must  challenge  PMs  to  use  a  business  case  development 
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methodology  to  support  M&S  investment  decisions.  Decision-makers  and  their  staff 
need  adequate  training  in  order  to  properly  implement  business  case-based  M&S 
investment  strategy  justification.  Additionally,  training  in  the  development  of  strategic 
management  techniques  and  practices  would  ensure  that  the  vision  step  of  the  business 
case  strategy  is  developed  from  an  issue  perspective  with  realistic  objectives. 

When  implementing  the  business  case  process,  I  recommend  several  points  be 
kept  in  mind  to  ensure  success.  Detailed  baselines  must  be  developed  with  future  data 
collection  efforts  in  mind.  The  assumptions  should  include  estimated  future  workload, 
useful  life  of  the  investment,  and  the  period  of  time  over  which  alternatives  will  be 
compared.  The  data  collection  activities  will  be  used  to  justify  and  compare  programs 
properly.  Additionally,  metrics  to  be  used  throughout  the  program  should  be  realistic  and 
tied  to  the  program  visions  and  goals.  Total  Ownership  Cost  (TOC)  must  be  addressed 
more  rigorously  for  all  programs  from  concept  through  disposal.  Too  often  TOC  is 
ignored  or  inadequately  addressed  during  the  analysis.  Rigorous  reviews  of  the  M&S 
effort  must  be  performed  throughout  the  program  with  continuous  feedback  to  all 
elements  of  the  program  being  developed  and  the  process  outputs  recorded  for  the  benefit 
of  future  development  programs.  The  output  of  the  Post  Implementation  R.eview  (PIR) 
should  feed  into  annual  performance  plans  which  will  support  the  vision  and  direction  set 
forth  in  the  master  or  strategic  plan  developed.  The  annual  performance  plans  should 
show  how  all  of  the  functional  requirements  and  the  feasibility  of  the  cost  and  benefits  of 
the  alternative  will  be  tied  to  program  schedules  and  budgets. 

Future  VPG  program  and  project  efforts  need  to  be  based  on  a  detailed  plan  which 
is  embraced  by  all  levels  of  management,  which  is  supported  by  an  annual  performance 
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plan,  and  outlines  the  metrics  by  which  the  individual  projects  will  be  assessed.  The 
performance  plan  should  tie  all  project  requirements  to  a  funding  line  in  the  budget.  All 
future  major  VPG  projects  which  support  the  development  of  the  DTC  stand-alone 
capability  and  those  that  are  tied  to  acquisition  programs  should  be  required  to  follow  the 
business  case  analysis  contained  herein  to  ensure  that  the  development  effort  is  cost 
effective  and  meets  the  technical  and  business  goals  of  DTC. 
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APPENDIX  A.  M&S  AND  CONTRIBUTING  WORLD 
TECHNOLOGY  EVENTS 


Year 

M&S  and  Contributing  World  Technology  Events  (In  Chronological 
Order) 

1867 

Rigid  Kriegspiel 

Ruggles  Orientator 

Sand  Tables 

1920  thru 

Tactical  Research  (Lindell  Hart) 

1930 

First  Link  Trainer 

1930  thru 

1940 

Wargames  at  Naval  War  College 

Navy  Buys  Instrumented  Trainer  ($1500) 

Army  Buys  6  link  trainers  @  $3500  each 

German  Soldiers  stand  guard  with  broom  sticks  simulating  rifles 

Turing  describes  “universal  machine” 

1940  thru 

1950 

Louisiana  Maneuvers 

Bleetchley  Park 

University  of  Illinois  Aviation  Lab 

Colossus  Computing  Machine  (cryptography) 

Eniac  (University  of  Reimsylvania) 

LINK  Foxmded 

Whirlwind  Digital  computer 

Operations  Research  Formalized  -  Eisenhower  Letter 

Transistor  Patented 

1950  thru 

1960 

Contact  Analog  Display  (George  Hoover,  USN) 

Link  begins  development  of  first  simulator  for  fighter  aircraft 

AAI  and  Texas  Instruments  Founded 

Flat  See  through  display  for  aircraft 

ILLIAC  (University  of  Illinois) 

First  commercially  available  computer  (Univac  1) 

Live/Virtual  Simulations  of  Air  Defense 

FAA  Air  Traffic  Control  Center  for  Simulations  (FAA/RAND/SDC) 

First  COE  As 

TI  Produces  Silicon  Transistor 

Navy  disposes  of  WMI  aircraft  -  Generally  Trainers 

DEC  Foimded 

FORTRAN 

SAGE  System 

Kwajalein  Test  Site  Selection 

Minsky  &  McCarthy  form  AJ  Department  at  MIT 
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First  Mini  Computer  PDP-1 

LISP  1.5 

1960  thru 
1970 

First  Triangulated  Irregular  Networked  Terrain 

IBM  7070 

ALGOL  60 

Line  of  Sight  Algorithms 

NATO  Range  Study 

Lockheed  Combat  Model 

Carmonette  Development 

Early  Digital  Terrain 

Teletype  Model  33  with  Punched  Paper  tape 

Simscript  (RAND) 

FORTRAN  IV 

Model  Board  Visual  Systems 

Image  Generator  for  NASA  Docking  Simulation  (GE) 

Navy  Transfers  Kwajalein  to  Army 

First  Mouse 

BASIC 

Lunar  Lander  Simulation  (GE) 

TACOS  Model  (Air  Defense) 

First  Commercial  Nuclear  Power  Plant  Simulation  (Indian  Point) 

First  Head  Mounted  Display  (Ivan  Sutherland) 

Evans  &  Sutherland  Founded 

SIMULA 

Computer  Image  Generator  (CIG)  for  Navy  (GE) 

First  Laser  Application  for  Direct  Fire  Weapons 

Dyntacs  Development 

Land  Combat  Systems  Study  LCS-90  using  Carmonette 

AF  Human  Resources  Lab  Aircrew  Research  Established 

SAIC  Founded 

ARPA  Begins  Development  of  A^ANET 

1970  thru 

1980 

First  Advanced  Simulator  for  Undergraduate  Pilot 

First  Non-Realtime  Demo  of  CIG  Potential  for  Airline  Trtdning 
(Hancock  Airport) 

Na\'y  Establishes  Top  Gun 

SEL86 

Air  Combat  Maneuvering  Range  and  Instrumentation  (ACMR/ACMI) 

AMC  71  Ground  Mobility  Model 

TACWARS  Development  Begins 

Royal  Navy  School  of  Training  Established 

Intel  Founded 

ATOWL  Mission  Planning  Software 

TETAM  Intervisibility  Experiment 
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LCOM  (TAC/LOG  Maiming) _ 

First  Micro  Processor  Chip  (Intel  4004  60,000  operations  per  sec  - 

$200) _ 

First  Email  Demo  _ 

Concepts  Evaluation  Model 
First  USAF  Helicopter  Simulation 

AMSWAG _ 

LCMS  (Log  Capability  Measurement  System) _ 

ATARI  Founded  and  ships  “PONG” _ _ 

C  Prolog _ 

Air  Force  Establishes  Red  Flag _ 

Mil-Std-1558  (Defacto  6-Post  Motion  System  Standard) 

CATTS  (Combined  Arms  Tactical  Training  System) 

DIVWAG 

First  Electronic  Calculator  (TI) _ 

Packet  Networking  Developed 

Intel  8080 _ 

Computer  Aided  Operations  Research  Facility  -  Kings  Point 

Two  Networked  Full  Motion  Wide  FOV  Flight  Simulations _ 

ARI  Report  on  Tactical  Engagement  Simulation  (TES) 

DIVLES _ 

Cope  Thunder  Exercise  (TI) _ 

Gates  &  Allen  Form  Microsoft _ 

First  Ethernet  At  Xerox _ 

Simulator  Fidelity  Studies  (Beginning  with  motionAVithout  motion) 
AFHRL 

AIMVAL/ACEVAL  Test 
MINI-3  (LLNL) 

VECTOR 

TACSM  (Artillery) _ 

Jobs  &  Wozniak  Form  Apple  and  Produce  Apple  1 
TRS-80 

NATO  Reference  Mobility  Model _ 

Initial  Concept  for  Large  Scale  Distributed  Simulator  Networking 


CIG  Dome  (VTRS) 

PTOS  (Patriot  Air  Defense  Simulation) 

ART  BASS  Studies 

AFOTEC  uses  simulation  to  identify  T&E  issues  and  extrapolate  results 
JANUS  (LLNL) 

TAC  BRAWLER  Development  Begins _ 

TRS-80  Calculates  Battle  Damage  at  Naval  War  College 

“War  in  Europe”  Game _ 

Reforger  ’79  driven  by  manxial  game  “battle” _ 

Motorola  68000 
DEC  11/780 
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_ Intel  8086 _ 

_  Table  Top  Gunnery  Trainer 

1980  thru  AMRAAM  Test  (four  linked  simulations) 

1990 _ 

_ First  Class  Cockpit  Simulator  (CAE  Link8767  &  MD80) 

_ Army  National  Training  Center  (NTC) _ 

_ First  PC  based  Constructive  Simulation  (Wang  2000) 

_  VIGS  (Video  Disc  Based  Gunnery) 

_  Tank  Driver  Simulator 

Navy  Mobile  Sea  Ranges 
'  IBM  PC 

_  Hayes  Smart  Modem  200 

Common  Lisp  ~  ~ 

_  Four  Low  Cost  Tank  Gunnery  Trainers  Networked 

_ VCASS  Helmet  Mounted  Display  _ 

_ ARPA  Video  Arcade  Tank  Gunnery  Trainer  ($.25  per  play) 

_  Mission  Training  Plans 

JANUS  delivered  to  Army 

_  Silicon  Graphics  Formed 

Geometry  Engine  (SGI) 

SUN  1 

CRAYXMP 
Postcscript 
TCP/IP  Adopted 
ADA 

_  Lotus  123 

_  Delta  Graphics  Founded 

_  Warrior  Preparation  Center  Established 

_  CAMMS  (Munition  in  a  minicomputer) 

_  ACA  Buy  (Urban  Game) 

_ Casteforem  Development  ~~ 

_ ARPA  SIMNET  —  Concept  Validation  Study 

_  MODSIM  (Modular  design  for  simulations) 

Compuscene  III  with  Texture  (GE) 

Air  Force  Established  Blue  Flag 
_  JANUS  4.0 

_ “Fire  in  the  East”  Board  Game _ 

_  Apple  Eye  (Intervisibility  on  a  microcomputer) 

VPL  Founded  (Jason  Lanier) 

MAC  128  “ 

_  ARPANET  Split  -  MILNET  Added  (Later  DDN) 

_  1,000  Most  Computers  on  the  net 

Apple  LaserWriter 

Motorola  68020  ~ 

FATS  Foimded 
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EOSAEL _ 

Bl-B  Engineering  Research  Simulation _ 

IPFN  Testbed  (ALBQ)  netted  to  PTOS  simulation  (El  Paso)  via  T1 


JRTC  Development  and  Demo 

Navy  Fallon  Tactical  Aircrew  Combat  Trainer  System  (TACTS) 

METT-CAP  Mission  Planner 

JESS 

VIC  (Vector  in  Commander) 

CBS  Development  Begins 

Battle  Command  Training  Program  (BCTP)  established  using  JESS 
Delta  Graphics  acquired  by  BBN 


Motorola  68030 

NSFNET  Created 

PERL 

Intel  80386 

JANUS  TRAC 

_ SEES  1.0  (LLNL) _ 

_ Air  Force  Red  Flag  measurement  and  debrief  system _ 

_  CMTC  Opens 

_ IRIS  4D70G  vdth  MIPS  R2000  (First  Unix  RISC  workstation  (SGI) 

_  TAC  Evaluation  of  Limited  FOV  Visual  (F15) 

Semi-Automated  Forces  (SAF)  Development  Begins 
First  Networked  Simulations  -  Ft.  Knox 
First  Z  Buffered  Lo'w  Cost  Multi-Channel  CIG  (Delta  Graphics) 
Multigen  1 .0 
~  SIMNET-D 

_ SIMNET  Graphics  Engine  (Delta  Graphics) _ 

Compuscene  PT  2000  (GE) 

_  OPTEC  FOG-M  Comparative  Test 

MS  Excel 

C-H-  2.0  (multiple  inheritance)  . 

_  10,000  Most  Computers  on  the  Net 

“Morris  Worm”  Attacks  7,000  of  60,000  Most  computers  on  the  Net 
~  NSF  Backbone  to  T1  (1.5  MB/sec) 

Viewpoint  Data  Labs  Founded 

_  Strategic  Defense  Simulation  and  Performance  Assessment 

_  Brave  Shield  “88  -  First  JWC  Distributed  CBS  Exercise 

EADSIM  Development  Begins 

First  Formal  User  Test  of  SIMNET  (Ft.  Knox  to  Hood)  60  Netted 
Simulations 

_  Distributed  Interactive  Simulation  (PIS)  Workshops  Begin 

_ First  Long  Haul  Training  Exercise  (Ft.  Knox  to  Rucker) _ 

SIMNET  Testbed,  250  Simulations,  1 1  Sites 
1990  thru  BFIT  (SIMNET  to  Naval  Ship) 

1995 _ 
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VP  1000  First  PC  Based  CIG  (TSI) 

Extended  Air  Defense  Test  Bed  (EADTB)  Begins 

Multiship  R&D  Testbed 

Marksmanship  Trainers  at  100  locations  (USMC) 

British  Army  Approves  Simulation  Strategy 

Crossbow  SA-12  Digital  Simulation  Prototype 

Simulation  Policy  Study  (OSD/FMP/TRF) 

ARPANET  Transitioned  (Ceases) 

Visual  Basic 

Sim  Sparc  2 

DMSO  Established 

Maritime  ITEM  Development  Begins 

First  Aggregate  Level  Simulation  Protocol  (ALSP)  Demo 

JCM  (LLNL/JWC) 

JMASS  Established 

Unit  Performance  Assessment  System  (UPAS) 

RADSIM  1.0  (Low  Cost  Radar  Simulation) 

CD  ROM  Map  Libraries 

SIMNETSAF6.1 

NPSNET 

Mobile  SIMNET  (ANG) 

First  DIS  1.0  Demon  (TSI) 

Close  Combat  Tactical  Trainer  (CCTT)  ASARC 

Battlefield  Distributed  Simulation  (BDS-D)  and  Advanced  Distributed 
Simulation  Tactical  (ADST-I) 

ALSP  ULCHI  Focus  Lens  Europe-US-Korea 

First  Executive  Council  Modeling  and  Simulation  (EXCIMS)  meeting 

MIPS  R4000  64  Bit  RISC 

NSFNET  BackBone  to  T3  (45  MB/sec) 

GOPHER  ( University  of  Minnesota) 

BICM  (Battlefield  Intel  Collection  Model) 

Defense  Science  Board  (DSB)  Report  on  Advanced  Distributed 

Simulation  (ADS) 

World  Wide  Web  (WWW)  application  (CERN) 

STRICOM  Established 

CSSTSS  Development  Begins 

Reforger  ’92  First  Simulation  Driven  FTX 

ALSP  Program  Funded 

JCM/SOFNET  (LLNL/JWFC  AF/DMSO) 

SEES  (Security  Exercise  Evaluation  System 

Marksmanship  Trainer  On  Board  (USMC) 

First  C-17  ATS 

Compuscene  VI  Target  Generation  (GE) 

MODSAF 

VRLink  Ships 
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SIMNETto  CSRDF 

CCTT  AWARD 

IITSEC  Exhibit  Floor  “On  the  Net” 

Warbreaker 

Joint  Precision  Strike  Demonstration  (JPSD) 

SGI  Reality  Engine 

First  Onyx  (SGI) 

Topscene  on  Aircraft  Carrier 

Automated  Training  analysis  and  Feedback  System  (ATAFS) 

Simulation  Training  Integrated  Performance  Evaluation  System 
(STRIPES) 

JMASS  1.0 

Core  Battle  Simulation  (CBS)  STAARS 

Joint  WarFighting  Center(JWFC)  Standup 

BCBST  (Battle  Command  and  Brigade  Staff  Trainer)  (BBS  and 

JANUS) 

Simulation  Demo  to  SASC 

Intel  60  MHZ  Pentium 

1,000,000  Host  Computers  on  the  NET 

MOSAIC  “Surf  the  NET” 

WWW  Worms,  Crawlers,  Snakes,  Spiders  collect  information  around 
the  WEB 

Motorola  Power  PC  601 

Martin  Marietta  Buys  GE  Aerospace 

Loral  Buys  BBN  Sim  +  Quintron 

AF/^OM  Established 

GPS  Based  Ranges 

AFRES  (Multi-Task  Trainer  Low  Cost  F-16  unit  level) 

USAF  Visual  System  Evaluation 

Synthetic  Theater  of  War  (STOW) 

ADS  Architecture  Study 

ZERO  Regard 

CFOR  CSSIL 

MDT2  Multi  Service  Distributed  Trainer  Testbed 

DIS  Protocol  i.O  Approved  (IEEE  1278) 

INCOMMS  ’94  First  Demo  of  Dismounted  unit  (virtual)  netted  to 
brigade  (constructive)  -  Ft.Benning 

TACCSF 

First  DIS  Voice  communications  (TSI) 

A2ATD  (Air  Defense  and  Armor) 

TADSIM  (Theater  wide  DIS  R&D) 

Infoscope  (TSI) 

JCOS 

Leathemet  (USMC) 

Zealous  Pursuit 
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IPORT  1.0  (Unicycle) 

High  Level  Architecture  (HLA)/Architecture  Management  Group 
(AMG) 

STOW -Europe 

MAIS-DIS  Compliancy  Demo 

MORS  ORSA  Handbook 

MORS  SimVal  Symposium 

Joint  ADS  JT&E  (JADS)  -  T&E  Evaluation 

JCM/SIMNET  (STRICOM) 

Atlantic  Resolve 

JTLS  1.85  (First  Joint  Multi-sided  Simulation) 

BIDS 

Loral  Buys  IBM  Federal  Systems 

MAC  Power  PC 

DEC  Alpha  300  Mhz 

GPS  Full  IOC 

Netscape  IPO 

7,000,000  Host  computers  &  61,000  Networks  on  the  NET 

VRML 

Windows  ‘95 

Lockheed  and  Martin  Marietta  Merge 

DMSTTIAC  Established 

DoD  M&S  Master  Plan 

BIDS  (LUT/DPG/UT) 

NTF  Wargame 

Roving  Sands 

JWID  ‘95 

Kernel  Blitz 

JPSD  ‘95 

Prairie  Warrior 

MOBA  Digital  Data  (Benning  &  Quantico) 

MILES  2000 

Small  Arms  Naval  Trainer 

HLA  Protofederations 

IPORT  2.0  (Treadmil) 

BBS-MODSAF 

ADST-II 

Table  1.  M&S  and  Contributing  World  Technology  Events  after  (Thorpe,  1995) 
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APPENDIX  B.  SIMULATION  BASED  ACQUISITION  (SBA) 

“CHEAT  SHEET” 


Tasking:  Lt  Gen  Martin  suggested  that  SARDA  develop  a  “cheat  sheet”  that  he 
and  other  decision-makers  can  use  to  ask  program  managers  the  “right  questions.”  The 
purpose  of  this  is  two-fold:  first,  to  educate  the  decision  makers  on  what  the  philosophy 
of  SBA  actually  means  when  it  comes  to  implementing  it  on  programs,  and  second,  to 
cause  the  program  managers  to  “answer  the  question  before  its  asked”  (as  Lt  Gen  Martin 
said)  and  in  effect,  start  implementing  SBA  on  their  programs  in  the  process  of  preparing 
to  answer  the  questions.  Following  are  (1)  general  modeling  and  simulation  questions, 
and  (2)  SBA  -  specific  modeling  and  simulation  questions. 

A.  GENERAL  M&S  QUESTIONS 

1.  M&S  Planning 

•  IS  THERE  A  PLAN  FOR  M&S  USE  ACROSS  THE  SYSTEM  LIFECYCLE? 

•  IS  THE  M&S  STRATEGIC  PLAN  INCLUDED  IN  THE  ACQUISITION 
STRATEGY  DOCUMENT?  IS  IT  UPDATED  ANNUALLY? 

•  IS  THE  PLAN  FUNDED? 

•  DOES  THE  PLAN  TAKE  INTO  CONSIDERATION: 

•  IDENTIFICATION  OF  ALL  MODEL  AND  SIMULATIONS 

•  THE  SOURCE  OF  NEEDED  DATA  FOR  THE  MODELS  AND  SIMULATIONS 

•  REUSE  ACROSS  ACQUISITION  PHASES? 

•  VV&A? 

•  USE  OF  MODELS  IN  SOURCE  SELECTION  AND  THEIR  RELEASE  TO 
INDUSTRY? 

•  IS  THE  USE  OF  M&S  ACCOUNTED  FOR  IN  THE  TEMP? 


2.  Requirements 


•  WHAT  MODELS/SIMS  AND  DATA  WERE  USED,  IF  ANY,  AS  PART  OF  THE 
AOA  TO  GENERATE  REQUIREMENTS?  WERE  THEY  VERIFIED,  VALIDATED 
AND  ACCREDITED? 


3.  Cost 


•  DOES  THE  FUNDING  PROFILE  SUPPORT  THE  ROBUST  USE  OF  M&S 
THROUGHOUT  THE  PROGRAMS  LIFECYCLE?  (E.G.,  FRONT-LOADED?) 

•  DOES  THE  USE  OF  M&S  MAKE  THE  PROGRAM  MORE  AFFORDABLE?  IF 
SO,  IN  WHAT  WAY? 

•  COST  AVOIDANCE 

•  COST  SAVINGS 


109 


•  SCHEDULE  SAVINGS 

•  OTHER 

4.  Contractor  Incentives 

•  ARE  THE  PROPER  INCENTIVES  IN  PLACE  FOR  THE  CONTRACTOR  TO  USE 
A  DIGITAL  ENVIRONMENT  TO  CONTAIN  COSTS? 

B.  SBA-SPECIFIC  QUESTIONS 

SBA  is  a  new  acquisition  strategy  that  exploits  recent  advances  in  M&S 
technologies.  It  involves  not  only  changes  in  the  technologies  and  environment,  it  also 
involves  changes  in  the  culture  and  processes.  The  following  questions  seek  evidence  of 
those  changes. 

SBA  enables  a  more  iterative  acquisition  process  versus  the  traditional  serial 
process  of  moving  through  the  phases.  We  conceive,  design,  manufacture,  test,  train, 
operate  &  sustain  (in  an  iterative  nature)  concurrently  in  the  virtual  world  before  locking 
in  on  a  solution.  We  delay  locking  in  our  requirements  (ORD)  until  we  examine  the  trade 
space  in  a  virtual  system  of  systems  environment.  As  the  system  is  developed  and 
produced,  the  representation  of  the  new  system  continues  to  be  matured  in  parallel  with 
that  development.  In  terms  of  implementation,  this  process  should  be  reflected,  in  part,  in 
the  program  acquisition  strategy  in  a  section  entitled  “M&S  approach”: 

1.  Shared  Digital  Environment  -  SBA  Concepts  That  Apply 

•  Collaborative  Environments  —  Simply  put,  this  is  2  or  more  programs  collaborating 
together  on  a  given  problem  and  sharing  and  reusing  models/sims.  Example:  missile 
PM  collaborating  with  aircraft  and  ship  PMs  for  a  strike  warfare  collaborative 
environment;  or,  several  aircraft  PMs  collaborating  across  the  aircraft  product  line 

•  QUESTION:  IS  THE  PROGRAM  PARTICIPATING  IN  ANY  COLLABORATIVE 
ENVIRONMENTS? 

•  Distributed  Product  Description  or  DPD  (e.g.,  virtual  representation  of  the  system 
including  all  the  associated  information  that  makes  it  “smart”  such  as  the  function  of  the 
system,  special  mzinufacturing  requirement,  cost  of  system,  etc.  a  DPD  is  a  step  beyond 
a  virtual  prototype) 

•  QUESTION:  DOES  THE  ACQUISITION  STRATEGY  CALL  FOR 
DEVELOPMENT  OF  A  DPD?  IF  SO,  WILL  THE  GOVERNMENT  OWN  THE 
RIGHTS?  IS  IT  PLANNED  TO  EVOLVE  THE  DPD  OVER  THE  LIFE  OF  THE 
PROGRAM? 

•  DoD/Industry  Resource  Repository  or  DIRR  -  A  shared  repository  for  dpds  as  well  as 
other  models  and  simulations  to  enable  government  and  industry  sharing  and  reuse.  It 
is  an  extension  of  the  existing  DoD  modeling  and  simulation  resource  repository 
(MSRR).  Access  controls  required  -  example:  Boeing  may  have  a  Boeing-proprietary 
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DPD  of  the  JSF  that  only  Boeing  and  the  government  can  access  (but  not  Lockheed). 
JSF  may  also  use  a  “piuple  DPD”  that  anyone  can  access. 


•  QUESTION:  DOES  THE  ACQUISITION  STRATEGY  CALL  FOR  PLACEMENT 
OF  THE  DPD  IN  THE  MSRR  (FUTURE  -DIRR)?  HAVE  APPROPRIATE  ACCESS 
CONTROLS/PROPRIETARY  DATA  RIGHTS  ISSUES  BEEN  RESOLVED? 

•  Ultimately,  when  SBA  is  implemented  as  a  strategy,  programs,  working  in 
conjunction  with  each  other,  will  use  common  standards  and  adopt  a  common 
architecture  which  will  facilitate  interoperability  and  reuse  supporting  DoD  goals  of 
reduction  in  cycle  time  and  cost. 


•  QUESTION:  HOW  IS  THE  PROGRAM  WORKING  TO  LEAVERAGE  EXISTING 
STANDARDS  (E.G.,  HLA)? 

•  QUESTION:  HOW  ARE  YOU  USING  STANDARDS  TO  ENSURE 

INTEROPERABILITY  WITH  OTHERS  OUTSIDE  YOUR  PROGRAM? 

•  QUESTION:  WHAT  M&S  ARE  YOU  LEAVERAGING  FROM  OTHER 
PROGRAMS? 

•  QUESTION:  WHAT  PARTNERSHIPS  HAVE  YOU  INTIATED/INTEND  TO 
SUSTAIN  TO  BOTH  CONTRIBUTE  TO  THE  COMMUNITY  AND  REDUCE 
DEPARTMENT  ACQUISITION  COSTS? 

2.  Simulation  Test  and  Evaluation  Process  (STEP) 


•  STEP  is  an  integral  part  of  the  SBA  strategy.  The  STEP  Guidelines,  signed  by  Dr. 
Sanders  and  Mr.  Coyle,  require  an  evaluation  strategy  integrating  test  and  evaluation 
(T&E)  with  M&S. 

•  QUESTION:  DOES  YOUR  PROGRAM  HAVE  AN  INTEGRATED  T&E  AND 
M&S  STRATEGY? 


•  QUESTION:  TO  WHAT  EXTENT  IS  YOUR  EVALUATION  STRATEGY 
IMPLEMENTING  THE  STEP  GUIDELINES? 

•  QUESTION:  STEP  REQUIRES  EARLY  INPUT  FROM  THE  TEST  COMMUNITY 
AS  PART  OF  THE  ACQUISITION  LIFE  CYCLE.  AS  PART  OF  YOUR 
EVALUATION  STRATEGY,  AT  WHAT  POINT  DO  THE  TESTERS  BECOME 
INVOLVED  IN  YOUR  PROGRAM? 

•  QUESTION:  WILL  OT  USE  M&S  FOR  lOT&E? 
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3.  Evolved  Process 


•  Integrated  Product  And  Process  Development  (IPPD)  and  the  use  of  Integrated 
Product  Teams  (IPTs)  with  empowered  government  and  industry  members  to 
accomplish  defined  tasks  are  critical  to  take  advantage  of  the  technological  advances 
which  make  SBA  possible. 


•  QUESTION:  HAS  THE  PROGRAM  FORMED  GOVERNMENT/INDUSTRY 
IPTS,  INCLUDING  ONE  FOR  M&S,  WITH  EMPOWERED  GOVERNMENT  AND 
INDUSTRY  MEMBERS  TO  ACCOMPLISH  THE  DEFINED  TASKS  THAT  ARE 
CRITICAL  TO  TAKE  ADVANTAGE  OF  THE  TECHNOLOGICAL  ADVANCES 
WHICH  MAKE  SBA  POSSIBLE? 

4.  Evolved  Culture 

•  Incentives  are  required  for  development  in  the  initial  stages  of  a  program  to  develop 
the  integrated  data  environment  and  other  infirastructure  and  resources  for  SBA. 


•  QUESTION:  HAS  THE  PROGRAM  MANAGER  PROVIDED  INCENTIVES  FOR 
INDUSTRY  TO  EITHER  ASSIST  IN  OR  DEVELOP  THE  NECESSARY 
PRODUCTS  AND  SERVICES  FOR  SBA  TO  BE  IMPLEMENTED  AS  A 
STRATEGY? 

•  The  process  must  build  a  partnership  between  government  and  industry.  The  first 
opportunity  of  this  is  during  the  request  for  proposals  and  proposal  evaluation 


•  QUESTION:  DOES  THE  ACQUISITION  STRATEGY  CALL  FOR  THE  SHARING 
OF  MODELS  &  SIMULATIONS  EARLY  ON  (ALA  IPPD),  TO  INCLUDE  DURING 
SOURCE  SELECTION,  AND  DURING  THE  LIFE  OF  THE  PROGRAM? 


(SARDA,  2000) 
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APPENDIX  C.  COST  FACTOR  DEFINITIONS 


A.  COMMUNICATIONS  -  NONRECURRING 

Total  nonrecurring  expenditure  of  communications  equipment  and  services  to 
make  the  fully  configured  and  installed  system  operable  at  its  inception.  Include  in  this 
category  data  communications  equipment,  such  as  modems  and  data  encryption  devices, 
as  well  as  other  communications  costs,  such  as  local  area  networks. 

B.  COMMUNICATIONS -RECURRING 

Yearly  payments  for  communications  costs.  Include  in  this  category  data 
communications  equipment,  such  as  modems  and  data  encryption  devices,  as  well  as 
other  communications  costs,  such  as  local  area  networks. 

C.  CONTRACTING 

Total  expenditure  of  contracts  for  construction,  design,  development,  consulting, 
and  installation  of  system. 

D.  CONVERSION  COSTS 

For  replacement  or  upgrade  systems,  the  incremental  costs  incurred  only  from  the 
costs  to  convert  hardware  (such  as  PCs,  mainframes,  disk  drives,  servers,  printers)  or 
software  (such  as  database,  batch  programs,  and  expert  systems)  from  one  system  to 
another.  These  costs  are  only  incurred  if  the  system  under  consideration  is  replacing 
another  specific  system.  Examples  are  batch  transfer  programs,  data  re-entry,  hardware 
modification. 

E.  EQUIPMENT 

Yearly  costs  allocated  for  the  purchase,  lease  or  rental  of  other  equipment  to 
support  business  operations  excluding  equipment  associated  with  the  new  system. 
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F.  EQUIPMENT  -  OTHER 

Yearly  costs  allocated  for  the  purchase,  lease  or  rental  of  other  equipment  to 
support  the  use  of  the  system.  Include  in  this  category  photocopiers,  file  cabinets,  fire 
safes,  microfiche,  optical  storage  facilities  and  other  office  products. 

G.  FACILITY  SPACE  OCCUPANCY 

Yearly  payments  for  allocated  rents  and  building  user  costs  for  business 
operations  other  than  the  space  allocated  to  the  new  system  (excludes  utilities). 

H.  HARDWARE  -  NONRECURRING 

Total  dollar  expenditure,  by  year,  of  hardware  to  make  the  fully  configured  and 
installed  system  operable  at  its  design  inception.  Include  in  this  category  mainframes, 
desktop,  laptop,  PCs,  disk  drives,  tape  drives,  display  monitors,  keyboards,  printers  and 
other  peripheral  equipment.  This  does  not  include  possible  conversion  costs  for 
upgrading  from  older  systems. 

I.  HARDWARE  LEASE  OR  RENTAL  -  RECURRING 

Yearly  payments  for  system  hardware  lease  or  rental.  Include  in  this  category 
mainframes,  desktop,  laptop,  PCs,  disk  drives,  tape  drives,  display  monitors,  keyboards, 
printers  and  other  peripheral  equipment. 

J.  INCIDENTALS 

Other  minor  costs  associated  with  day  to  day  business  operations.  Exclude 

incidental  costs  associated  with  the  new  system. 

K.  INTERAGENCY 

Yearly  payments  to  (less  credits  received  from)  other  agencies  for  shared  facilities 
used  and  services  other  than  systems  operations. 
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L. 


MAINTENANCE 


Yearly  payments  for  maintenance  of  business  operations.  Exclude  from  this 
category  all  repairs,  maintenance,  and  emergency  service  support  costs  directly 
attributable  to  the  system. 

M.  PARALLEL  OPERATIONS 

Expenditure  needed  for  parallel  operations  or  systems  testing  on  a  nonrecurring 

basis. 

N.  PERSONNEL  SALARIES  AND  FRINGE 

Yearly  allocated  costs  for  persoimel  who  perform  Benefits  functions  to  be 
supported  by  the  new  system.  This  should  include  all  organizational  units  and  sites  whose 
operations  will  change  as  a  result  of  the  system. 

O.  RESIDUAL  VALUE 

The  salvage  value  of  the  entire  system  at  the  conclusion  of  its  life  cycle. 

P.  SECURITY 

Yearly  costs  to  provide  system  security  and  integrity.  Include  in  this  category 
security  monitoring  systems,  alarm  systems,  camera  and  voice  recording  and  storage 
systems,  lock  and  pass  key  systems,  and  security  personnel  costs. 

Q.  SOFTWARE -NONRECURRING 

Total  dollar  expenditure  of  software  to  make  the  fully  configured  and  installed 
system  operable  at  its  inception.  Include  in  this  category  all  system  software  packages, 
off-the-shelf  software,  and  custom  software,  site  or  network  license  and  all  original 
software  programming  and  development  costs.  Important:  do  not  include  software 
conversion  or  upgrade  costs  here;  instead,  include  these  in  conversion  costs. 
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R.  SOFTWARE  LEASE  OR  RENTAL  -  RECURRING 

Yearly  payments  for  system  software  lease  or  rental.  Include  in  this  category  all 
system  software  packages,  off-the-shelf  software,  and  custom  software,  site  or  network 
license,  and  all  original  software  maintenance  costs. 

S.  STUDIES 

Total  expenditure  for  studies  necessary  to  implement  this  system  alternative. 

T.  SUPPLIES 

Yearly  payments  for  supplies  to  operations  other  than  those  associated  with  the 
new  system. 

U.  SYSTEM  TESTING  AND  BACKUP 

Recurring  costs  to  test  reliability  and  integrity  of  system  in  operation,  including 
costs  for  memory  back  up.  This  does  not  include  costs  for  nonrecurring  start-up  and 
installation  testing. 

V.  TRAINING 

Recurring  training  costs  associated  with  routine  business  operations,  e.g.,  training 
aimed  at  improving  supervisory  skills.  Exclude  costs  for  training  associated  with  the  new 
system. 

W.  TRAVEL 

Yearly  travel  expenses  incurred  in  the  normal  course  of  business  operations. 
Exclude  any  travel  costs  that  are  associated  with  the  new  system.  Include  train,  bus,  taxi, 
and  airline  tickets,  gas,  mileage  and  toll  charges  and  auto  rental  expenses. 

X.  USEFUL  SYSTEM  LIFE 

Forecast  of  the  planned  useful  system  life  from  the  first  month  of  system 
implementation. 
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Y.  UTILITIES 

Yearly  payments  for  costs  of  utilities  allocated  to  business  operations  other  than 
the  new  system. 
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APPENDIX  D.  DOCUMENTATION  CHECKLIST 


DOCUMENTATION  CHECKLIST 

□  Alternatives  Analysis: 

□  Benehts  Cost  Analysis  (BCA): 

•  Answer  the  Three  Tesky  Ques^ins. 

•  Assumptions  and  constraints 

•  Alternatives  and  fiieir  schedule,  costs,  and  benefits 

1)  Does  Ihe  investment  in  a  major  capital  asset  support 

«  Risk  and  sensitivity  analysis 

corc/priority  mission  functions  that  need  to  be  perfonned  by 

«  Performance  goals  and  measures  for  monitoring  the 

the  Federal  GovemmOTt? 

project 

2)  Does  the  investment  need  to  be  undertaken  by  the 
requesting  agency  because  no  alternative  private  sector  or 

□  Feasibility  Analysis  and  Market  Research: 

govemmeot  source  can  better  sujjportthe  function? 

•  Feasibility: 

♦  Availability 

3)  Does  the  investment  support  work  processes  tiiat  have  been 

•  Affordable 

simplified  or  otherwise  redesigned  to  reduce  cost,  improve 
effectiveness,  and  make  maximum  use  of  commercial,  off-thc- 

•  Cost  &  Benefits 

self  technology? 

•  Market  Research: 

•  Avail^ility  of  commercial  items  to  meet  the  need, 
and  whether  they  require  modification 

□  Project  Plan: 

•  Distribution  and  support  ci^)abililics  of  suppliers 

•  Major  milestones  with  schedule 

•  Deliverables 

Q  Functional  Requirements  Analysis: 

•  Critical  path  decision  points 

•  Resources 

•  Performance  criteria,  goal,  or  ultimate  ouqjut 

•  Integrated  Project  Team 

•  Definition  of  the  common  uses  of  the  IT  investment 

•  Ranking  of  each  requirement  in  order  of  importance 

•  Decon^sition  of  functional  requirements  into  self- 

□  Acquisition  Plan  -  Part  One: 

contained  features 

•  Statement  of  need 

•  Requirements  should  be  described  in  terms  of: 

•  Significant  conditions  affecting  acquisition 

♦  Business  outcome 

•  Capability  or  performance 

•  Purpose 

•  Tradeoffs 

•  S/SO  program  components  involved 

*  Risks 

•  Operating  constraints 

•  Modular  contracting 

•  Mission 

•  Capd>ility 

□  Baseline  Assessment  and  Statement  of  Need: 

•  Schedule  and  cost  objectives 

•  Baseline  assessment  criteria 

•  Functionality 

□  QAPlan: 

•  Full  life  <^lc  cost 

•  Life  cycle  process 

•  E3q)ccted  funding  levels  cost 

•  Security  Plan 

♦  Periodic  independent  review 

•  Capacity  to  manage  asset 

•  Existing  resource  baseline 

□  Risk  Management  Plan: 

•  Performance  gap 

•  Risks  including  costs,  schedule  and  technical 

•  Potential  risks 

Figure  6.  Document  Checklist  from  (GSA,  2000) 
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DOCUMENTATION  CHECKLIST  (CONT.) 

□ 

SecurttvPjlan; 

□ 

PjSASTER/RECOVERY  AND  CONTINGENCY  PLAN: 

* 

Govenunent  information  protected  from  misuse^  loss  and 

• 

Strategy  for  mission  performance  and  recovery  from 

unauthorized  access 

loss  of  existing  support 

• 

Security  responsibilities  assigned 

» 

Plans  for  continuous  testing  of  the  system 

• 

Systems  security  plan  reviewed  by  security  specialist 

* 

Rules  of  the  system 

□ 

Post  IMEIJEMJENTATION  REVIEW: 

• 

Training  issues 

• 

Incident  response  capabilities 

* 

System  reviewed  and  documented  on  a  regular  basis 

• 

Intcrconnectivitysccurity  addressed 

m 

PIR  conducted  by  an  indcpcDdent  evaluation  team 

Security  controls  reviewed 

0 

Customer  user  satisfaction  reviewed 

0 

Internal  business  reviewed 

□ 

Acquisition  Plan  -  Part  Two: 

0 

Strategic  impact  and  effectiveness  reviewed 

0 

Innovation  reviewed 

• 

Source  of  supply 

• 

Conyctition  description 

• 

Source  selection  procureincnt 

• 

Contracting  consideration 

• 

Budgeting  and  funding 

• 

Product  description 

1 

• 

Priorities 

• 

Contractor  vs.  Government  preference 

• 

Pcrfonnance  Management  System 

Test  and  evaluation 

• 

Logistics  considerations 

» 

Indicate  Government  furnished  property 

• 

Indicate  Government  furnished  tnfomiatioa 

□ 

Plans  roR  Assets  f  IT  Investments: 

• 

Operational  analysis 

• 

Steady  State  Plan 

• 

Schedule  Post  Implementation  Review 

• 

Asset  Disposal  Plan 

Figure  7.  Document  Checklist  Continued  from  (GSA,  2000) 
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APPENDIX  E.  PROJECT  RISKS  CHECKLIST 


PROJECT  RISKS  CHECKLIST 


□  Strategic  Risk 

Alignment  with  the  agency's  overall  business  strategy 
Qarity  of  expression  of  anticipated  project  outcomes. 

Presence  of  metrics  to  verify  &e  successful  completion  of  each  project  phase. 

□  Financial  Risk 

Size  of  expenditure  required. 

Existence  of  cost/benefit  analysis. 

Existence  of  defined  payback  and  time  frame  of  payback. 

Reputation  and  financid  status  of  vendor(s). 

□  Project  Management  Risk 

Experience  of  project  management  teams 
Existence  of  woiic  plan  for  entire  life  cycle. 

Degree  of  development  of  measurable  milestones. 

Length  of  time  for  project  implementation. 

Existence  of  system  for  tracking  unresolved  issues. 

Definition  of  user  and  development  skill  requirements. 

□  Technology  Risk 

Plan  for  validating  that  user  needs  are  met. 

Existence  of  load  test  in  accordance  with  industry  standards 
Evaluation  of  technology  options. 

Maintainability  and  ability  to  upgrade  key  technologies. 

Vendor's  ability  to  implement  technology. 

□  Security  Risk 

Perform  risk  assessment 
Implement  security  controls. 

Serarity  training  and  awareness. 

Contingency  planning  &  disaster  recovery. 

Comply  with  security  policy. 

□  Change  Management/Operationa]  Risk 

Development  of  acceptance  plan. 

Experience  and  ability  of  existing  staff  to  support  new  system. 

Organization's  frmiliarity  with  proposed  hardware/software  environment 
Development  of  system  operating  procedures. 

Inq^act  to  organization  of  system  failure. 

Magnitude  of  change  introduced  by  system. 

Number  of  business  units  impacted. 


Figure  8.  Risk  Checklist  from  (GSA,  2000) 
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